Image processing method, device, storage medium and electronic device
By smoothing the image features output from the image processing network, the problem of poor airspace continuity in image processing is solved, and the natural transition effect of the image is improved.
Patent Information
- Application Number
- CN202010749780.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-07-30
AI Technical Summary
The existing image processing methods have poor continuity in the airspace, resulting in unnatural transitions of the processed image in the local area adjacent pixels.
By smoothing the image features output by the image processing network, the specific method is to smooth the image features of each element and its adjacent elements to obtain the airspace smooth image features, and generate the processed image based on the smoothing features.
The continuity of the processed image in the airspace is improved, the pixel value difference is reduced, and the natural transition effect of the image is improved.
Smart Images

Figure CN114066737B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to an image processing method, device, storage medium and electronic device. Background Art
[0002] The related technologies of image processing often involve using a pre-built image processing network to extract image features from the image to be processed, and then generating a processed image based on the image features of the image to be processed.
[0003] In order to improve the efficiency of image processing network in extracting image features and reduce the computational complexity of image processing network, in image processing techniques involving image processing network, the image to be processed is often downsampled to obtain a downsampled image with lower resolution, and then the image processing network is used to extract image features from the downsampled image.
[0004] The processed images generated by the existing processing methods often have the problem of poor continuity in the spatial dimension (i.e., spatial domain), where the poor continuity in the spatial domain is manifested as the transition between adjacent pixels in a local area of a processed image being inconsistent with the original image to be processed. Summary of the invention
[0005] The present disclosure provides an image processing method, device, storage medium and electronic device to at least solve the problem of poor continuity of processed images in the spatial domain in the related art. The technical solution of the present disclosure is as follows:
[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:
[0007] Get the image to be processed;
[0008] Inputting the image to be processed into an image processing network to obtain image features of the image to be processed;
[0009] Smoothing each element of the image feature of the image to be processed and the adjacent elements of the element to obtain a spatial domain smoothed image feature of the image to be processed; the adjacent elements of the element include: elements located within a preset range centered on the element in the image feature of the image to be processed;
[0010] A processed image corresponding to the image to be processed is generated according to the spatial domain smoothed image features of the image to be processed.
[0011] Optionally, in an implementation, after inputting the image to be processed into an image processing network to obtain image features of the image to be processed, the method further includes:
[0012] Performing smoothing processing on each element of the image feature of the image to be processed and the corresponding element of the element to obtain the time-domain smoothed image feature of the image to be processed; the corresponding element of the element includes: the element at the same position as the element in the time-domain smoothed image feature of a previous image adjacent to the image to be processed in the time domain, or the element at the same position as the element in the image feature of a previous image adjacent to the image to be processed in the time domain;
[0013] Wherein: the step of smoothing each element of the image feature of the image to be processed and its adjacent elements to obtain the spatial domain smoothed image feature of the image to be processed includes: smoothing each element of the temporal domain smoothed image feature of the image to be processed and its adjacent elements to obtain the spatial domain smoothed image feature of the image to be processed.
[0014] Optionally, in an implementation, after smoothing each element of the image feature of the image to be processed and its adjacent elements to obtain the spatial domain smoothed image feature of the image to be processed, the method further includes:
[0015] Smoothing each element of the spatial domain smoothed image feature of the image to be processed and the corresponding element of the element to obtain the temporal domain smoothed image feature of the image to be processed; the corresponding element of the element includes: the element at the same position as the element in the temporal domain smoothed image feature of a previous image adjacent to the image to be processed in the temporal domain, or the element at the same position as the element in the spatial domain smoothed image feature of a previous image adjacent to the image to be processed in the temporal domain;
[0016] Wherein: generating a processed image corresponding to the image to be processed according to the spatial domain smoothed image features of the image to be processed includes: generating a processed image corresponding to the image to be processed according to the temporal domain smoothed image features of the image to be processed.
[0017] Optionally, in one implementation, the performing smoothing on each element of the image feature of the image to be processed and its adjacent elements to obtain the spatial domain smoothed image feature of the image to be processed includes:
[0018] Determining the size of the smoothing kernel of the feature map according to the attribute of the feature map contained in the image feature;
[0019] Aligning each element of each feature map of the image feature with the center of the smoothing kernel, and updating each of the elements to the arithmetic mean or weighted mean of all elements covered by the smoothing kernel;
[0020] The set of all updated feature maps is used as the spatial domain smoothed image feature of the image to be processed.
[0021] Optionally, in an implementation, determining the size of the smoothing kernel of the feature map according to the attribute of the feature map included in the image feature includes:
[0022] Determining the size of the smoothing kernel of the feature map according to a comparison result of the size of the feature map, a preset maximum smoothing kernel size, and a preset minimum smoothing kernel size;
[0023] Alternatively, the gradient value of the feature map is calculated, and the size of the smoothing kernel of the feature map is calculated according to the gradient value of the feature map.
[0024] Optionally, in an implementation, the performing smoothing on each element of the image feature of the image to be processed and the corresponding element of the element to obtain the time-domain smoothed image feature of the image to be processed includes:
[0025] The image features of the image to be processed and the time-domain smoothed image features of the previous image or the image features of the previous image are weighted averaged to obtain the time-domain smoothed image features of the image to be processed.
[0026] Optionally, in one implementation, the smoothing process is performed on each element of the spatial domain smoothed image feature of the image to be processed and the corresponding element of the element to obtain the temporal domain smoothed image feature of the image to be processed, including:
[0027] The spatial domain smoothed image features of the image to be processed and the temporal domain smoothed image features of the previous image or the spatial domain smoothed image features of the previous image are weighted averaged to obtain the temporal domain smoothed image features of the image to be processed.
[0028] Optionally, in one implementation, the method further includes:
[0029] In response to the image feature of the image to be processed having a similarity with the time-domain smoothed image feature or the image feature of the previous image being not less than a similarity threshold, performing weighted averaging of the image feature of the image to be processed and the time-domain smoothed image feature of the previous image or the image feature of the previous image to obtain the time-domain smoothed image feature of the image to be processed;
[0030] In response to the image feature of the image to be processed having a similarity with the time-domain smoothed image feature or the image feature of the previous image being less than a similarity threshold, the image feature of the image to be processed is used as the time-domain smoothed image feature of the image to be processed.
[0031] Optionally, in one implementation, the method further includes:
[0032] In response to the spatial domain smoothed image feature of the image to be processed having a similarity with the temporal domain smoothed image feature or the spatial domain smoothed image feature of the previous image being not less than a similarity threshold, weighted averaging the spatial domain smoothed image feature of the image to be processed and the temporal domain smoothed image feature of the previous image or the spatial domain smoothed image feature of the previous image is performed to obtain the temporal domain smoothed image feature of the image to be processed;
[0033] In response to the spatial domain smooth image features of the image to be processed, the similarity between the spatial domain smooth image features and the temporal domain smooth image features or the spatial domain smooth image features of the previous image is less than a similarity threshold, and the spatial domain smooth image features of the image to be processed are used as the temporal domain smooth image features of the image to be processed.
[0034] Optionally, in one implementation, generating a processed image corresponding to the image to be processed according to the spatial domain smoothed image features of the image to be processed includes:
[0035] Sum all feature maps contained in the spatial domain smooth image feature of the image to be processed to obtain a smooth feature map, and perform coefficient interpolation on the smooth feature map to obtain the image enhancement feature of the image to be processed; wherein the size of the image enhancement feature of the image to be processed is consistent with the size of the image to be processed;
[0036] For each pixel of the image to be processed, the pixel value of the pixel is adjusted according to the element corresponding to the pixel in the image enhancement feature of the image to be processed to obtain the adjusted pixel, and all the adjusted pixel points are combined into a processed image.
[0037] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:
[0038] An acquisition unit, configured to acquire an image to be processed;
[0039] A processing unit, configured to input the image to be processed into an image processing network to obtain image features of the image to be processed;
[0040] A first smoothing unit is configured to perform smoothing processing on each element of the image feature of the image to be processed and the adjacent elements of the element to obtain a spatial domain smoothed image feature of the image to be processed; the adjacent elements of the element include: elements located in a preset range centered on the element in the image feature of the image to be processed;
[0041] The generating unit is configured to generate a processed image corresponding to the image to be processed according to the spatial domain smoothed image features of the image to be processed.
[0042] Optionally, in one implementation, the method further includes:
[0043] A second smoothing unit is configured to perform smoothing processing on each element of the image feature of the image to be processed and the corresponding element of the element to obtain the time-domain smoothed image feature of the image to be processed; the corresponding element of the element includes: the element at the same position as the element in the time-domain smoothed image feature of a previous image adjacent to the image to be processed in the time domain, or the element at the same position as the element in the image feature of the previous image adjacent to the image to be processed in the time domain;
[0044] Wherein: when the first smoothing unit performs smoothing processing on each element of the image feature of the image to be processed and the adjacent elements of the element to obtain the spatial domain smoothed image feature of the image to be processed, it is configured to: perform smoothing processing on each element of the time domain smoothed image feature of the image to be processed and the adjacent elements of the element to obtain the spatial domain smoothed image feature of the image to be processed.
[0045] Optionally, in one implementation, the method further includes:
[0046] A third smoothing unit is configured to perform smoothing processing on each element of the spatial domain smoothed image feature of the image to be processed and the corresponding element of the element to obtain the temporal domain smoothed image feature of the image to be processed; the corresponding element of the element includes: the element at the same position as the element in the temporal domain smoothed image feature of a previous image adjacent to the image to be processed in the temporal domain, or the element at the same position as the element in the spatial domain smoothed image feature of a previous image adjacent to the image to be processed in the temporal domain;
[0047] Wherein: when the generating unit generates a processed image corresponding to the image to be processed according to the spatial domain smoothed image features of the image to be processed, it is configured to: generate a processed image corresponding to the image to be processed according to the temporal domain smoothed image features of the image to be processed.
[0048] Optionally, in an implementation, the first smoothing unit includes:
[0049] a determining unit configured to determine a size of a smoothing kernel of the feature map according to an attribute of the feature map contained in the image feature;
[0050] An updating unit, configured to align each element of each feature map of the image feature with the center of the smoothing kernel, and update each of the elements to an arithmetic mean or a weighted mean of all elements covered by the smoothing kernel;
[0051] The obtaining unit is configured to use the set of all updated feature maps as the spatial domain smoothed image features of the image to be processed.
[0052] Optionally, in one implementation, the determining unit includes:
[0053] A first determining subunit is configured to determine the size of the smoothing kernel of the feature map according to a comparison result of the size of the feature map, a preset maximum smoothing kernel size, and a preset minimum smoothing kernel size;
[0054] Alternatively, the second determining subunit is configured to calculate the gradient value of the feature map, and obtain the size of the smoothing kernel of the feature map according to the gradient value of the feature map.
[0055] Optionally, in an implementation, the second smoothing unit includes:
[0056] The first weighted averaging unit is configured to perform weighted averaging on the image features of the image to be processed and the time-domain smoothed image features of the previous image or the image features of the previous image to obtain the time-domain smoothed image features of the image to be processed.
[0057] Optionally, in one implementation, the third smoothing unit includes:
[0058] The second weighted averaging unit is configured to perform weighted averaging on the spatial domain smoothed image features of the image to be processed and the temporal domain smoothed image features of the previous image or the spatial domain smoothed image features of the previous image to obtain the temporal domain smoothed image features of the image to be processed.
[0059] Optionally, in one implementation, the method further includes:
[0060] A first control unit is configured to control the first weighted averaging unit to perform weighted averaging of the image feature of the image to be processed and the time-domain smoothed image feature of the previous image or the image feature of the previous image in response to the image feature of the image to be processed having a similarity with the time-domain smoothed image feature of the previous image or the image feature of the previous image being not less than a similarity threshold, so as to obtain the time-domain smoothed image feature of the image to be processed;
[0061] It is also configured to respond to the image feature of the image to be processed, and the similarity with the time-domain smoothed image feature or the image feature of the previous image is less than a similarity threshold, and use the image feature of the image to be processed as the time-domain smoothed image feature of the image to be processed.
[0062] Optionally, in one implementation, the method further includes:
[0063] a second control unit configured to control the second weighted averaging unit to perform weighted averaging of the spatial domain smoothed image feature of the image to be processed and the temporal domain smoothed image feature of the previous image or the spatial domain smoothed image feature of the previous image in response to the similarity between the spatial domain smoothed image feature of the image to be processed and the temporal domain smoothed image feature of the previous image or the spatial domain smoothed image feature of the previous image, so as to obtain the temporal domain smoothed image feature of the image to be processed;
[0064] It is also configured to respond to the spatial domain smooth image features of the image to be processed, and the similarity between the spatial domain smooth image features and the temporal domain smooth image features or the spatial domain smooth image features of the previous image is less than a similarity threshold, and use the spatial domain smooth image features of the image to be processed as the temporal domain smooth image features of the image to be processed.
[0065] Optionally, in one implementation, the generating unit includes:
[0066] A generating subunit is configured to sum all feature maps included in the spatial domain smooth image feature of the image to be processed to obtain a smooth feature map, and perform coefficient interpolation on the smooth feature map to obtain an image enhancement feature of the image to be processed; wherein the size of the image enhancement feature of the image to be processed is consistent with the size of the image to be processed;
[0067] The adjustment unit is configured to adjust the pixel value of each pixel of the image to be processed according to the element corresponding to the pixel in the image enhancement feature of the image to be processed, obtain the adjusted pixel, and combine all the adjusted pixel into a processed image.
[0068] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:
[0069] processor;
[0070] a memory for storing instructions executable by the processor;
[0071] The processor is configured to execute the instructions to implement an image processing method as provided in any one of the first aspects of the embodiments of the present disclosure.
[0072] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the image processing method provided by any one of the first aspect of the embodiments of the present disclosure.
[0073] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided. When the computer program product is executed, it is used to implement any one of the image processing methods provided in the first aspect of an embodiment of the present disclosure.
[0074] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0075] By smoothing each element of the image feature of the image to be processed and its adjacent elements, the spatial domain smoothed image feature of the image to be processed is obtained, and the elements in the obtained spatial domain smoothed image feature are close to the adjacent elements, thereby improving the continuity of the processed image generated according to the spatial domain smoothed image feature in the spatial domain.
[0076] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0077] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0078] Figure 1 is a schematic diagram showing a comparison of an image before and after processing by an existing image processing method according to an exemplary embodiment;
[0079] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment;
[0080] Figure 3 is a schematic diagram of two consecutive processed images in a video obtained by another existing image processing method according to an exemplary embodiment;
[0081] Figure 4 is a flowchart of another image processing method according to an exemplary embodiment;
[0082] Figure 5 is a flow chart of a method for processing image features using a spatial domain smoothing method according to an exemplary embodiment;
[0083] Figure 6 is a flow chart showing a method for processing image features using a time domain smoothing method according to an exemplary embodiment;
[0084] Figure 7 is a flowchart of another image processing method according to an exemplary embodiment;
[0085] Figure 8is a schematic diagram showing a comparison between an image processed by the image processing method provided by the present disclosure and an image before the processing according to an exemplary embodiment;
[0086] Fig. 9 is a schematic diagram of two consecutive processed images in a video obtained by another image processing method provided by the present disclosure according to an exemplary embodiment;
[0087] Fig.10 is a block diagram of a device for processing an image according to an exemplary embodiment;
[0088] Fig.11 is a block diagram of another apparatus for processing an image according to an exemplary embodiment;
[0089] Fig.12 is a block diagram of another apparatus for processing an image according to an exemplary embodiment;
[0090] Fig.13 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0091] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0092] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0093] The method proposed in the present disclosure can be applied to any image processing task that processes an image to be processed through a pre-trained image processing network, including but not limited to image enhancement, image stylization, image restoration, etc.
[0094] As described in the background technology, in the existing image processing methods implemented by using image processing networks, in order to reduce the amount of calculation of the image processing network, the image to be processed is often first downsampled to obtain the corresponding downsampled image, and then the downsampled image is input into the image processing network to obtain the image features of the image to be processed, and finally the image features of the image to be processed and the image to be processed are combined to generate the processed image. This method will cause the continuity of the processed image in the spatial domain to be worse than that of the original image to be processed.
[0095] Among them, poor spatial continuity can be manifested as unnatural transitions in certain areas of the processed image (or inconsistency with the original image to be processed). One manifestation of poor spatial continuity can be referred to Figure 1 . Figure 1 The left side is the image to be processed. Figure 1 The right side is the processed image obtained after stylizing the image to be processed. It can be found that the colors of the background pixels other than the buildings in the two rectangular boxes in the processed image are basically the same, while in the processed image, the background part has a significant unnatural boundary in the two rectangular boxes at the same position.
[0096] In view of the above problems, the present disclosure proposes an image processing method, which improves the continuity of the processed image generated according to the image features in the spatial domain by smoothing the image features output by the image processing network.
[0097] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 As shown, the image processing method includes the following steps.
[0098] In step S21, an image to be processed is obtained.
[0099] The above-mentioned image to be processed may include an independent static image, for example, a photo taken in a certain scene, or a picture drawn by a user using drawing software, both of which are independent static images.
[0100] The image to be processed may also refer to each image frame in a video to be processed. For example, if image enhancement is required for a video, assuming that the video contains 1,000 image frames, each of the image frames may be used as an image to be processed, and each image frame of the video to be processed may be processed one by one using the image processing method provided in any embodiment of the present disclosure, thereby obtaining a processed image corresponding to each image frame, and finally, all the processed images are combined in order according to the corresponding image frames in the video to be processed, thus forming a processed video after image enhancement.
[0101] In step S22, the image to be processed is input into an image processing network to obtain image features of the image to be processed.
[0102] The above-mentioned image processing network is a neural network that is pre-trained according to a specific image processing task using a corresponding plurality of training samples.
[0103] For example, if it is necessary to perform image enhancement on the image to be processed using the method provided in this embodiment, each training sample used to train the neural network includes an original image and an enhanced image obtained by performing image enhancement on the original image using other methods.
[0104] If it is necessary to perform stylized processing on the image to be processed using the method provided by this embodiment, each training sample includes an original image and a stylized image obtained by stylizing the original image using other methods.
[0105] The image features of the image to be processed output by the image processing network generally include N feature maps, where N is the number of channels of the image processing network, and each channel of the image processing network corresponds to a feature map. Each feature map can be considered as a feature matrix with W rows and H columns. The numbers that constitute each feature matrix are equivalent to the elements of the feature matrix (or feature map). W and H are preset positive integers, and their specific values are determined by the structural parameters of the image processing network, while the structural parameters of the neural network can be set based on experience when determining the structure of the image processing network. The elements of each feature map contained in the image features depend on the model parameters of the image processing network and the pixel values of each pixel in the image to be processed. For a trained image processing network, its model parameters are uniquely determined. Therefore, the elements of each feature map of the image features output by the image processing network depend on the pixel values of each pixel in the image to be processed. In other words, the values of the elements contained in the image features reflect the pixel values of the corresponding pixels in the image to be processed.
[0106] In step S22, the image processing network for extracting image features of the image to be processed may include several convolution layers, each convolution layer is provided with several channels, each channel corresponds to a convolution kernel, each convolution kernel can be considered as a matrix of M rows and M columns, M is a preset structural parameter of the image processing network, the value of each element in each convolution kernel is the model parameter that needs to be determined when training the image processing network. After the training of an image processing network is completed, the value of each element of each convolution kernel is uniquely determined.
[0107] Each convolution layer is used to perform convolution operations on the input using its own convolution kernel. The input of the first convolution layer can be the image to be processed or the downsampled image obtained after downsampling the image to be processed. The input of the second and subsequent convolution layers can be the output of the previous convolution layer or the output of the previous convolution layer after pooling by the pooling layer. The output of the last convolution layer is the image feature of the image to be processed output by the image processing network. The number of feature maps contained in the image feature is equal to the number of channels of the last convolution layer.
[0108] The image processing network used in the method provided in the present invention can be established based on the structure of any existing image processing network, and the method of training the image processing network using a large number of training samples can also refer to any existing image processing network training method, which will not be described in detail here.
[0109] Various image processing tasks, including image enhancement and image stylization, can be completed using trained image processing networks or other algorithms instead of image processing networks. Generally, the use of algorithms other than image processing networks often requires cyclic processing of the image to be processed, and finding an optimal result as the processed image through multiple cycles. However, when using image processing networks for processing, it is only necessary to input the image to be processed into the image processing network, and then calculate each convolution layer of the image processing network one by one to obtain the processed image. Therefore, in image processing tasks, using image processing networks to process images can significantly improve the efficiency of image processing, so that a large number of processed images corresponding to the images to be processed can be obtained in a short time.
[0110] In step S23, each element of the image feature of the image to be processed and its adjacent elements are smoothed to obtain the spatial domain smoothed image feature of the image to be processed.
[0111] In step S23, for any element of an image feature, the adjacent elements of the element include: elements located in a preset range centered on the element in the image feature. The size of the preset range can be determined according to the attributes of the feature map to which the element belongs during smoothing.
[0112] As described in step S23, smoothing is performed on each element of an image feature and its adjacent elements. This can be considered as smoothing the image feature using a spatial domain smoothing method.
[0113] The spatial domain smoothing method may be: for any element A in each feature map, obtain the value of element A and other elements in a certain range centered on element A in the feature map, then calculate the weighted average or arithmetic mean of all the obtained elements, and update the value of element A to the calculated weighted average or arithmetic mean, thereby completing the smoothing of element A based on the spatial domain smoothing method. By repeating the above process for each element in a feature map, the spatial domain smoothing feature map corresponding to the feature map can be obtained. The above process is performed on all feature maps of an image feature to obtain the spatial domain smoothing feature map corresponding to each feature map. The combination of all spatial domain smoothing feature maps is the spatial domain smoothing feature corresponding to the image feature.
[0114] Combination Figure 1 As shown in the comparison diagram, some local areas of the processed image have unnatural transitions (i.e., the spatial continuity is poor). This can be considered to be caused by the large difference between the pixel values of some pixels in the area and the pixel values of adjacent pixels in the processed image, while the original image to be processed does not have the problem of poor spatial continuity. Therefore, it can be considered that the large difference in pixel values of adjacent pixels in the local area is because the elements of the corresponding areas in the image features of the image to be processed output by the image processing network differ too much.
[0115] By updating the elements in each feature map of the image feature to the arithmetic mean or weighted mean of multiple adjacent elements within a certain range, each element in the spatial domain smoothing feature obtained after processing can be close to the adjacent elements to a certain extent. Therefore, compared with the image features directly output by the image processing network, the difference between each two adjacent elements in the spatial domain smoothing feature obtained after processing using the spatial domain smoothing method will be smaller than the difference between the corresponding elements in the image features not processed using the spatial domain smoothing method. Furthermore, in the processed image generated according to the spatial domain smoothing feature, the difference between the pixel values of each two adjacent pixels will also be smaller than the difference between the corresponding pixel values in the image generated according to the original image features. Reflected in the macroscopic image, the continuity of the processed image generated according to the spatial domain smoothing feature is better than that of the image generated according to the image features output by the image processing network that have not been processed by the spatial domain smoothing method.
[0116] In step S24, a processed image corresponding to the image to be processed is generated according to the spatial domain smoothed image features of the image to be processed.
[0117] The image processing method provided in this embodiment uses a spatial domain smoothing method to process the image features output by the image processing network to obtain corresponding spatial domain smoothed image features. Since each element in the spatial domain smoothed image features is the weighted average or arithmetic average of the element at the corresponding position in the original unprocessed image features and its adjacent elements. Compared with the unprocessed image features, each element in the spatial domain smoothed image features is closer to the adjacent elements of the element. Therefore, the continuity of the processed image generated according to the spatial domain smoothed image features will be closer to the original image to be processed, that is, compared with the processed image directly generated by the image features that have not been processed by the spatial domain smoothing method, it has better continuity in the spatial domain.
[0118] As mentioned above, the image processing method can be used to process a video to obtain a processed video composed of a number of consecutive frames of processed images. When the image processing network is used to process a video, on the one hand, the processed images constituting the processed video may have the aforementioned poor continuity problem in the spatial domain, and on the other hand, the processed video may also have the problem of poor continuity in the temporal domain when playing.
[0119] Please refer to Figure 3 When the existing processing method is used to perform stylized image processing on a number of consecutive image frames in a video, the following may occur: Figure 3 The phenomenon shown, Figure 3 The left side is a processed image obtained after the previous image frame in the video is processed, and the right side is a processed image obtained after the next image frame in the video is processed.
[0120] observe Figure 3 It can be found that the contents displayed in the two image frames before and after the original video are basically the same, while bright spots of different shapes appear in the rectangular frames at the same position of the two processed images. If the processed video contains bright spots of different shapes at different positions in each frame of the processed image, then the bright spots of different shapes will appear and disappear alternately when the processed video is played, causing the audience to observe flickering in some areas. This flickering that occurs during video playback is a manifestation of poor continuity in the time domain of the processed image.
[0121] In order to solve the problem of poor continuity in the time domain when processing a video, another embodiment of the present disclosure provides an image processing method, please refer to Figure 4 , the method may include the following steps:
[0122] In step S41, an image to be processed is obtained.
[0123] The purpose of this embodiment is to improve the problem of poor continuity in the time domain of the processed video obtained by processing using the image processing network. Therefore, the image to be processed in step S41 refers to each frame of the video to be processed. The process of processing the video to be processed using the method of this embodiment can be that first, the first frame of the video to be processed is used as the image to be processed, and the method provided by this embodiment is used to obtain the corresponding first frame of processed image, and then the method provided by this embodiment is used to process the second frame of the video to be processed to obtain the second frame of processed image, and so on. Finally, all the processed images are combined in order to obtain the processed video.
[0124] In step S42, the image to be processed is input into an image processing network to obtain image features of the image to be processed.
[0125] In step S43, the image features of the image to be processed are smoothed using a spatial domain smoothing method and a temporal domain smoothing method respectively to obtain smoothed image features of the image to be processed.
[0126] Optionally, in step S43, the image features of the image to be processed can be first smoothed using a spatial domain smoothing method to obtain the spatial domain smoothed image features of the image to be processed, and then the spatial domain smoothed image features of the image to be processed can be smoothed using a time domain smoothing method to obtain the time domain smoothed image features of the image to be processed, and the processed time domain smoothed image features are used as the smoothed image features of the image to be processed.
[0127] In this way, the execution process of step S43 may be, firstly, Figure 2 In the corresponding embodiment, the spatial domain smoothing method described in step S23 performs smoothing processing on the image features of the image to be processed to obtain the spatial domain smoothed image features of the image to be processed.
[0128] After obtaining the spatial domain smoothed image features of the image to be processed, the spatial domain smoothed image features of the image to be processed are smoothed using a temporal domain smoothing method to obtain the temporal domain smoothed image features of the image to be processed.
[0129] The method of smoothing the spatial domain smooth image features using the time domain smoothing method can be:
[0130] Each element of the spatial domain smoothed image feature of the image to be processed is smoothed with its corresponding element to obtain the temporal domain smoothed image feature of the image to be processed.
[0131] The corresponding element of an element in the spatial domain smooth image feature of the image to be processed may refer to the element located at the same position as the element in the temporal domain smooth image feature of the previous image adjacent to the image to be processed in the time domain, or may also refer to the element located at the same position as the element in the spatial domain smooth image feature of the previous image adjacent to the image to be processed in the time domain.
[0132] When the image to be processed is the first frame image in the video to be processed, the above-mentioned time domain smoothing method may not be executed, and the spatial domain smoothed image features of the image to be processed may be directly used as the time domain smoothed image features of the image to be processed.
[0133] Optionally, the image features of the image to be processed can be first smoothed using a time domain smoothing method to obtain the time domain smoothed image features of the image to be processed, and then the time domain smoothed features of the image to be processed can be smoothed using a spatial domain smoothing method to obtain the spatial domain smoothed image features of the image to be processed, and the spatial domain smoothed image features of the image to be processed can be used as the smoothed image features of the image to be processed.
[0134] The process of smoothing the image features of the image to be processed by using the time domain smoothing method may be: smoothing each element of the image features of the image to be processed with the corresponding element of the element to obtain the time domain smoothed image features of the image to be processed.
[0135] The corresponding element of an element in the image feature of an image to be processed may refer to the element located at the same position as the element in the time domain smoothed image feature of the previous image adjacent to the image to be processed in the time domain, or may refer to the element located at the same position as the element in the image feature of the previous image adjacent to the image to be processed in the time domain.
[0136] When the image to be processed is the first frame image in the video to be processed, the image features of the image to be processed can be directly used as the time domain smoothing features of the image to be processed.
[0137] The method of smoothing the time domain smoothing image features of the image to be processed by using the spatial domain smoothing method is similar to the above Figure 2 The corresponding embodiment has a similar step S23, which requires smoothing each element in the time-domain smoothed image feature and its adjacent elements. The spatial-domain smoothed image feature obtained after smoothing each element of the time-domain smoothed image feature is the smoothed image feature of the image to be processed, which is processed by the time-domain smoothing method and the spatial-domain smoothing method in sequence.
[0138] In step S44, a processed image is generated according to the smooth image features of the image to be processed.
[0139] Similar to the aforementioned spatial domain smoothing method, the time domain smoothing method disclosed in the present invention smoothes an element and its corresponding element by performing weighted averaging on the element and its corresponding element based on preset weights, and updating the element to the element obtained by the weighted averaging.
[0140] Further, in combination with the description of the elements and the corresponding elements of the elements in the previous text, the image features of the image to be processed are smoothed using the time domain smoothing method. It can be understood that the image features of the image to be processed and the image features of the image in the previous frame of the image to be processed in the video to be processed (or the time domain smoothed image features of the previous frame) are weighted averaged according to preset weights for every two elements located in the same position, and the weighted averaged elements are used as the elements at the corresponding positions in the time domain smoothed image features of the image to be processed. The above process can also be understood as weighted averaging the image features of the image to be processed and the image features of the image in the previous frame of the image to be processed in the video to be processed (or the time domain smoothed image features of the previous frame) according to preset weights, and the image features obtained after the weighted average are used as the time domain smoothed image features of the image to be processed.
[0141] Similarly, using a time domain smoothing method to smooth the spatial domain smoothed image features of the image to be processed can also be understood as taking a weighted average of the spatial domain smoothed image features of the image to be processed and the spatial domain smoothed image features of the image in the previous frame of the image to be processed in the video to be processed (or the time domain smoothed image features of the previous frame) according to preset weights, and using the image features obtained after the weighted average as the time domain smoothed image features of the image to be processed.
[0142] refer to Figure 3 Similar to the poor continuity in the spatial domain, the poor continuity in the time domain between the two adjacent processed images can also be considered to be caused by the large difference between the pixel values of the pixels at the same position in the two processed images. The large difference in the pixel values of the pixels in the two processed images is caused by the large difference between the elements at the same position in the image features of the two images.
[0143] By executing the above-mentioned time domain smoothing method on the image features of the images adjacent to each other in the processed video, the elements in the time domain smoothing features of the latter frame image in the processed video can be made close to the elements in the time domain smoothing features of the previous frame image, thereby reducing the difference in pixel values of the same position of the two adjacent processed frames generated according to the time domain smoothing features, thereby avoiding flickering when the processed images are played continuously.
[0144] Furthermore, since the image features of the image to be processed are processed using the time domain smoothing method and the spatial domain smoothing method respectively in step S43, the smoothed image features finally obtained have better continuity in the spatial domain and in the time domain compared to the image features directly output by the image processing network.
[0145] Please refer to the following Figure 5 , Figure 5 The present invention is a flowchart of a method for processing image features based on a spatial domain smoothing method according to an exemplary embodiment. The method may include the following steps:
[0146] In step S51, a feature map is extracted from the image features and the smoothing kernel size is determined.
[0147] As mentioned above, an image feature may include multiple feature maps (the number depends on the number of channels of the image processing network), and the spatial domain smoothing method is to smooth the adjacent elements in a feature map. Therefore, when there are multiple feature maps for an image feature, it is necessary to use the spatial domain smoothing method to process each of the feature maps one by one.
[0148] The smoothing kernel is a matrix that satisfies the following conditions:
[0149] The number of rows and columns are equal to the smoothing kernel size P, and P is an odd number greater than 1. All elements in the matrix are positive numbers, and the sum of all elements is equal to 1.
[0150] like Figure 5 As shown, the image features in the figure include four feature maps, and the smoothing kernel in the figure is a matrix with 3 rows and 3 columns, that is, the smoothing kernel size is 3.
[0151] In step S52, each element of the feature map is scanned using a smoothing kernel, and the scanned element is updated.
[0152] After each element in a feature map is scanned by the smoothing kernel and the corresponding value is updated, the spatial domain smoothing feature map corresponding to the feature map is obtained.
[0153] After scanning and updating a feature map, if there are unprocessed feature maps in the image features of the image to be processed, continue to execute steps S51 and S52 until the spatial smooth feature maps corresponding to all feature maps in the image features to be processed are obtained. The combination of all spatial smooth feature maps is the spatial smooth image features of the image to be processed.
[0154] In step S51, the smoothing kernel size may be determined based on any of the following three methods in combination with the attributes of the feature map.
[0155] The first method to determine the smoothing kernel size is to substitute the height (i.e., the number of rows) H and width (i.e., the number of columns) W of the feature map into the following formula (1):
[0156] Z=min[P1,max(P2,a×min(W,H))]
[0157] Wherein, a is a preset coefficient factor, P1 is a preset maximum smoothing kernel size, and P2 is a preset minimum smoothing kernel size. If the result Z of the above formula (1) is an odd number, Z is directly used as the smoothing kernel size P determined in step S51. If Z is not an odd number, an odd number not greater than Z and having the smallest difference with Z is determined as the smoothing kernel size P.
[0158] Combination Figure 5 , Figure 5 The number of rows and columns of the feature map in is 5, that is, W and H in the above formula are both 5.
[0159] The second method to determine the smoothing kernel size is:
[0160] First, the gradient value Y of the feature map is calculated using the following formula (2):
[0161]
[0162] In the above formula, X i,j Represents the element in the i-th row and j-th column of the feature map.
[0163] Formula (2) means that for each element except the elements in the last row and the last column of the feature map, the absolute value of the difference between the element and the first element below it, and the absolute value of the difference between the element and the first element to the right of it are calculated, the two absolute values are added and averaged, and the result is used as the gradient of the element.
[0164] Finally, the arithmetic mean of all gradients in the feature map is calculated, and the result is the gradient value of this feature map.
[0165] by Figure 5 Taking the feature map as an example, for element A11, the absolute value of the difference between A11 and A12, as well as the absolute value of the difference between the elements of A11 and A21 are calculated, and then the two are averaged to obtain the gradient of A11.
[0166] For each element except the last row (i.e. A51 to A55) and the last column (i.e. A15 to A45), the corresponding gradient can be calculated according to the above method, and finally the arithmetic mean of all gradients is calculated, and the result is Figure 5 The gradient value Y of the feature map in.
[0167] Then substitute the gradient value Y of the feature map into the following formula (3):
[0168] Z=min[P1, max(P2, b*Y / Mean)]
[0169] Wherein, a is a preset coefficient factor, P1 is a preset maximum smoothing kernel size, and P2 is a preset minimum smoothing kernel size. If the result Z of the above formula (1) is an odd number, Z is directly used as the smoothing kernel size P determined in step S51. If Z is not an odd number, an odd number not greater than Z and having the smallest difference with Z is determined as the smoothing kernel size P.
[0170] In formula (3), Mean is the mean of all elements of the feature map, and the calculation formula is shown in the following formula (4):
[0171]
[0172] A third method for determining the size of the smoothing kernel may be that, when a feature map is smoothed using the spatial domain smoothing method, a fixed smoothing kernel size may not be determined, but the scanned elements may be determined one by one, and then, for each scanned element, the gradient of the scanned element is calculated using the method for calculating the gradient of a single element in the feature map shown in the aforementioned formula (2), and then a suitable smoothing kernel size is determined based on the gradient of the scanned element, and then the scanned element is updated based on the elements covered by the smoothing kernel after the determined size.
[0173] That is to say, when a feature map is smoothed using a spatial domain smoothing method, the size of the smoothing kernel can be adjusted in real time according to the gradient of the scanned element.
[0174] When using the spatial domain smoothing method to smooth each feature map of the image feature, if the size of the smoothing kernel is too large, the elements of the entire feature map will tend to be the same, making the subsequent processing effect of the processed image generated according to the image features poor. If the size of the smoothing kernel is too small, the smoothing effect will be poor, so that the continuity of the processed image in the spatial domain cannot be effectively improved. Therefore, it is necessary to select a suitable gradient value according to the size or gradient value of the feature map so that the processed image has both a significant processing effect and good continuity in the spatial domain.
[0175] In step S52, using a smoothing kernel to scan an element in the feature map means aligning the central element of the smoothing kernel with the scanned element. At this time, the other elements of the smoothing kernel will be aligned with the other elements around the scanned element, that is, the smoothing kernel will cover the other elements around the scanned element.
[0176] Updating the scanned element means multiplying each element in the feature map covered by the smoothing kernel with the aligned element in the smoothing kernel, then accumulating all the products to obtain a value, and using the obtained value as the updated element.
[0177] If part of the smoothing kernel is outside the feature map, it can be considered that the element in the feature map corresponding to the element of this part of the smoothing kernel outside the feature map is 0.
[0178] When all elements of the smoothing kernel are equal, the above update process can be considered as calculating the arithmetic mean of all elements covered by the smoothing kernel in the feature map, and then updating the scanned elements to the calculated arithmetic mean. This process can be considered as performing mean filtering on all elements covered by the smoothing kernel in the feature map.
[0179] When all elements of the smoothing kernel are not completely equal, the above update process can be considered as using the corresponding elements in the smoothing kernel as weights, calculating the weighted average of all elements covered by the smoothing kernel in the feature map, and then updating the scanned elements to the calculated weighted average. This process can be considered as Gaussian filtering of all elements covered by the smoothing kernel in the feature map.
[0180] by Figure 5 For example, when scanning the element A11 in the feature map, the smoothing kernel covers the four elements A11, A12, A21, and A22. If mean filtering is performed, the four elements A11, A12, A21, and A22 can be summed, and then the sum is divided by 9. The final calculation result is the updated element, that is, B11 in the spatial smoothing feature map on the right. The process of scanning and updating element A44 is similar. A33, A34, A35, A43, A44, A45, A53, A54, and A55 are summed, and the sum is divided by 9. The final calculation result is B44 in the spatial smoothing feature map on the right.
[0181] From the above calculation process, it can be found that the updated elements contain part of the components of the adjacent elements in the original feature map (that is, other elements covered by the smoothing kernel during scanning). Therefore, it can be considered that compared with the original feature map, each element in the spatial smoothing feature map processed by the spatial smoothing method is close to the adjacent elements to a certain extent, so that the processed image generated by the spatial smoothing feature composed of multiple spatial smoothing feature maps has good continuity in the spatial domain, avoiding unnatural transitions in the processed image.
[0182] like Figure 4As described in the corresponding embodiment, when the image to be processed is a frame image in a video to be processed, the image features of the image to be processed can be smoothed using the spatial domain smoothing method, and then the spatial domain smoothed image features obtained by the spatial domain smoothing method can be smoothed using the temporal domain smoothing method to obtain the temporal domain smoothed image features of the image to be processed. In this case, the method of smoothing the image features using the spatial domain smoothing method is the same as Figure 5 The method shown is consistent.
[0183] In another case, the image features of the processed image are first smoothed using the time domain smoothing method to obtain the time domain smoothed image features, and then the time domain smoothed image features are smoothed using the space domain smoothing method to obtain the space domain smoothed image features. Figure 5 The object to be processed in the method shown, that is, the image features of the image to be processed, is replaced by the time domain smoothed image features of the image to be processed, and then executed Figure 5 The corresponding steps of the method shown can obtain the spatial domain smoothed image features after smoothing the temporal domain smoothed image features of the image to be processed using the spatial domain smoothing method.
[0184] Figure 6 is a flowchart of a method for processing image features using a time domain smoothing method according to an exemplary embodiment. Figure 6 ,The method for processing each image feature based on the time domain smoothing method includes:
[0185] It should be noted that since the time domain smoothing method will only be executed when the image processing method provided by the present invention is applied to process the video to be processed, in order to facilitate the understanding of the time domain smoothing method provided by the present invention, this embodiment takes a video to be processed consisting of multiple continuous frames of images to be processed as an example to illustrate the process of using the time domain smoothing method provided by the present invention to process the image features (or spatial domain smoothed image features) of each frame of the image to be processed in the video to be processed one by one.
[0186] In step S61, the image features of the first frame of the image to be processed are determined as the time-domain smoothed image features of the first frame of the image to be processed.
[0187] In step S62, the second frame of the image to be processed is determined as the current image to be processed.
[0188] In step S63, it is detected whether the image features of the current image to be processed meet the similarity condition.
[0189] The similarity condition of step S63 may be whether the similarity between the image feature of the current image to be processed and the time-domain smoothed image feature of the previous frame of the image to be processed is less than a preset similarity threshold.
[0190] If the similarity between the image features of the current image to be processed and the time-domain smoothed image features of the previous frame of the current image to be processed is greater than or equal to the preset similarity threshold, it is detected that the current image to be processed meets the similarity condition, and step S64 is executed.
[0191] If the similarity between the image features of the current image to be processed and the time domain smoothed image features of the previous frame of the current image to be processed is less than the similarity threshold, it is detected that the current image to be processed does not meet the similarity condition, and step S65 is executed.
[0192] In step S64, the image features of the current image to be processed and the time-domain smoothed image features of the previous frame of the current image to be processed are weighted averaged based on preset weights to obtain the time-domain smoothed image features of the current image to be processed.
[0193] In other optional embodiments of the present disclosure, step S64 may also be replaced by weighted averaging the image features of the current image to be processed and the image features of the previous frame of the image to be processed based on preset weights. In other words, when performing the weighted averaging described in step S64, the time domain smoothed image features obtained by smoothing the previous frame of the image to be processed using the time domain smoothing method are not used for smoothing, but the image features of the previous frame of the image to be processed that are not smoothed using the time domain smoothing method are directly used for weighted averaging.
[0194] If the weighted average is performed by using the image features of the previous frame of the image to be processed, the similarity condition in the above step S63 can be replaced by:
[0195] Whether the similarity between the image features of the current image to be processed and the image features of the previous frame of the image to be processed is less than a preset similarity threshold.
[0196] Assume that the current image to be processed is the t-th frame of the video to be processed, t is greater than or equal to 1, and less than or equal to the total number of frames contained in the video to be processed, the previous frame of the current image to be processed is the t-1-th frame, the time domain smoothed image feature of the image to be processed is recorded as CD, the image feature of the image to be processed is recorded as CN, and the subscripts are used to indicate the time domain smoothed image feature and image feature of the frame of the video to be processed, that is, the image feature of the t-th frame of the image to be processed is recorded as CN t , the time domain smoothing image feature of the t-th frame to be processed is recorded as CD t , the time domain smoothing image feature of the t-1th frame to be processed is recorded as CD t-1 , then step S64 can be expressed by the following formula (5):
[0197] CD t =Lam×CN t +(1-Lam)×CD t-1
[0198] Wherein Lam is the weight preset in step S64.
[0199] Furthermore, considering that the dimensions of the image features and the time-domain smoothed image features of any frame of the image to be processed are equal, that is, the image features and the time-domain smoothed image features each include K corresponding feature matrices (also called feature maps), each feature matrix includes H rows and W columns of elements, and CD t (k, i, j) represents the element of the i-th row and j-th column of the k-th feature matrix in the time-domain smoothed image feature of the t-th frame to be processed. Other image features are similar to the time-domain smoothed image feature. Then formula (5) can be written as the following formula (6):
[0200] CD t (k,i,j)=Lam×CN t (k,i,j)+(1-Lam)×CD t-1 (k,i,j)
[0201] Among them, the value range of k is a closed interval with an upper limit of K and a lower limit of 1, the value range of i is a closed interval with an upper limit of H and a lower limit of 1, and the value range of j is a closed interval with an upper limit of W and a lower limit of 1.
[0202] That is to say, step S64 means that, for the image features of the current image to be processed and the time-domain smoothed image features of the previous frame of the image to be processed, the elements belonging to the same position in these two features are weighted averaged to obtain the elements at the corresponding positions in the time-domain smoothed image features of the current image to be processed.
[0203] In step S65, the image feature of the current image to be processed is determined as the time-domain smoothed image feature of the current image to be processed.
[0204] In step S66, if the current image to be processed is not the last frame of the image to be processed, the next frame of the image to be processed is determined as the current image to be processed.
[0205] After step S66 is executed, the process returns to step S63 until the current image to be processed is the last image to be processed before step S66 is executed. In other words, the process obtains the time domain smoothed image features of each frame of the image to be processed in the video to be processed.
[0206] Step S66 refers to determining the next frame of the image to be processed after the current image to be processed before executing step S66 as a new current image to be processed.
[0207] refer to Figure 6 If the current image to be processed is the second frame of the video to be processed before executing step S66, then when executing step S66, the next frame after the second frame, that is, the third frame of the image to be processed is determined as the current image to be processed; if the current image to be processed is the third frame of the video to be processed before executing step S66, then when executing step S66, the next frame after the third frame, that is, the fourth frame of the image to be processed is determined as the current image to be processed, and so on.
[0208] For the aforementioned step S64, when obtaining the time-domain smoothed image features of the current image to be processed, it is also possible to consider not only the current image to be processed and the previous frame of the image to be processed of the current image to be processed, but also to further detect whether the next frame of the image to be processed of the current image to be processed satisfies the similarity condition. If the next frame of the image to be processed of the current image to be processed satisfies the similarity condition, then step S64 can be changed to weighted averaging of the image features of the current image to be processed, the image features of the next frame of the image to be processed of the current image to be processed, and the time-domain smoothed image features of the previous frame of the current image to be processed, that is, a total of three image features based on preset weights, and using the obtained result as the time-domain smoothed image features of the current image to be processed.
[0209] After obtaining the time-domain smoothed image features of each frame of the image to be processed in the video to be processed, if the time-domain smoothed image features of the image to be processed do not need to be processed by the spatial domain smoothing method in the subsequent process, the time-domain smoothed image features of each frame of the image to be processed can be directly determined as the smoothed image features of this frame of the image to be processed; if the time-domain smoothed image features of each frame of the image to be processed need to be processed by the spatial domain smoothing method in the subsequent process, the spatial domain smoothed image features obtained after the spatial domain smoothing method is processed are determined as the smoothed image features of each frame of the image to be processed.
[0210] For any two adjacent frames of to-be-processed images in the to-be-processed video, if the image features of the latter frame of the to-be-processed image meet the similarity condition, then it can be considered that the contents of the two frames of to-be-processed images are highly similar. By executing the above-mentioned time domain smoothing method on the image features of every two adjacent and highly similar frames of to-be-processed images in the to-be-processed video, each element in the time domain smoothed image features of the latter frame of the to-be-processed image can be made to approach the element at the same position in the time domain smoothed image features of the previous frame of the to-be-processed image, thereby enhancing the continuity in the time domain of the two adjacent processed images generated according to the time domain smoothed image features, and avoiding or reducing the abnormal flickering of the processed video obtained by combining the processed images during playback.
[0211] In step S63, when detecting whether the similarity between the image features of the current image to be processed and the time-domain smoothed image features of the previous frame of the image to be processed is less than the similarity threshold, the current image to be processed is recorded as the t-th frame of the image to be processed in the video to be processed, and the marking of the previous step S64 is used. The similarity between the image features of the current image to be processed and the time-domain smoothed image features of the previous frame of the image to be processed can be calculated using the following formula (7):
[0212]
[0213] Among them, D t It represents the similarity between the image features of the current image to be processed and the time-domain smoothed image features of the previous frame of the image to be processed.
[0214] Optionally, considering that the above similarity may not accurately represent the similarity between the image features of the current image to be processed and the time-domain smoothed image features of the previous frame of the image to be processed, the above similarity may be divided by the normalization coefficient Dg corresponding to the current image to be processed. t , get the normalized similarity D t / Dg t , use the normalized similarity and similarity threshold for comparison, and determine whether the image features of the current image to be processed meet the similarity conditions based on the comparison results.
[0215] Normalization coefficient Dg corresponding to the current image to be processed t The calculation formula is shown in the following formula (8):
[0216] Dg t =Sig×D t-1 +(1-Sig)×Dg t-1
[0217] That is to say, the normalization coefficient Dg corresponding to the current image to be processed t , which is equal to the normalization coefficient Dg of the previous frame of the image to be processed t-1 The similarity D corresponding to the previous frame of the image to be processed t-1 The weighted average of , where Sig is the preset weight and Sig is a real number less than 1.
[0218] For the first frame of the image to be processed in the video to be processed, its corresponding normalized coefficient Dg1 similarity D t All are set to 1.
[0219] like Figure 6 The method shown can not only directly process the image features of each frame of the image to be processed, but also process each feature map of the image features of each frame of the image to be processed separately.
[0220] For example, assuming that each image feature has K feature maps, then, Figure 6 The method shown may be, first, determining the first feature map of the image features of the first frame of image to be processed as the first time-domain smoothed feature map of the first frame of image to be processed, and then, starting from the second frame of image to be processed, detecting one by one whether the similarity between the first feature map of the image features of each frame of image to be processed and the first time-domain smoothed feature map of the time-domain smoothed image features of the previous frame of image to be processed is less than a similarity threshold.
[0221] For an image to be processed whose similarity is greater than or equal to a similarity threshold, a weighted average is performed on the first feature map of the image features of the frame to be processed and the first time-domain smoothed feature map of the time-domain smoothed image features of the previous frame to be processed of the frame, and the result of the weighted average is used as the first time-domain smoothed feature map of the time-domain smoothed image features of the frame to be processed.
[0222] After obtaining the first time-domain smooth feature map of each frame of the video to be processed, the above method is repeated to obtain the second time-domain smooth feature map, the third time-domain smooth feature map, and so on for each frame of the video to be processed, until each time-domain smooth feature map of each frame of the video to be processed is obtained, and then the multiple time-domain smooth feature maps of the same frame of the image to be processed are combined into the time-domain smooth image feature of the frame of the image to be processed.
[0223] When this method is used, the above formula (6) can be rewritten as formula (9):
[0224] CD t,k (i,j)=Lam×CN t,k (i,j)+(1-Lam)×CD t-1,k (i,j)
[0225] That is, the elements in the kth feature map of the image feature of the current image to be processed and the elements at the corresponding positions in the kth feature map of the time-domain smoothed image feature of the previous frame to be processed are weighted averaged to obtain the elements at the corresponding positions in the kth time-domain smoothed feature map of the time-domain smoothed image feature of the current image to be processed.
[0226] Similarly, formula (7) can be rewritten as the following formula (10):
[0227]
[0228] Formula (10) is used to use Figure 6When the time domain smoothing method shown in the figure processes the kth feature map in the image features of each frame of the image to be processed one by one, the similarity D between the kth feature map of the image features of the current image to be processed and the kth time domain smoothed feature map of the time domain smoothed image features of the previous frame of the image to be processed is calculated. t,k .
[0229] Correspondingly, for the kth feature map, execute Figure 6 When the method shown in the figure is used, the normalization coefficient is also changed to Dg t,k , and its calculation formula is as follows (11):
[0230] Dg t,k =Sig×D t-1,k +(1-Sig)×Dg t-1,k
[0231] Formula (11) indicates that for the kth feature map, Figure 6 The method shown in the figure, the normalization coefficient Dg of the current image to be processed on the kth feature map t,k Equal to the similarity D of the image features of the previous frame to be processed on the kth feature map t-1,k and the normalization coefficient Dg of the image feature of the previous frame to be processed on the kth feature map t-1,k Similarly, the normalization coefficient and similarity of the image features of the first frame to be processed on each feature map are set to 1.
[0232] It should be noted that if Figure 6 The method for processing image features based on the time domain smoothing method shown can also be used to process spatial domain smoothed image features obtained after being processed using the spatial domain smoothing method.
[0233] Will Figure 6 The method shown is used to process the spatial domain smoothed image features obtained after processing using the spatial domain smoothing method, that is, when processing the video to be processed, the image features of the image to be processed contained in the video to be processed are first smoothed using the spatial domain smoothing method to obtain the corresponding spatial domain smoothed image features, and then the spatial domain smoothed image features of the image to be processed are subjected to the temporal domain smoothing method. Figure 6 The steps in the embodiment shown can be replaced as follows:
[0234] In the aforementioned embodiment, step S61 may be replaced by determining the spatial domain smoothed image features of the first frame of the image to be processed as the temporal domain smoothed image features of the first frame of the image to be processed.
[0235] Step S63 may be replaced by detecting whether the spatial domain smoothed image feature of the current image to be processed satisfies a similarity condition. The similarity condition may be:
[0236] It may be whether the similarity between the spatial domain smoothed image features of the current image to be processed and the temporal domain smoothed image features of the previous frame of the image to be processed is less than a preset similarity threshold.
[0237] Step S64 can be replaced by weighted averaging the spatial domain smoothed image features of the current image to be processed and the temporal domain smoothed image features of the previous frame of the current image to be processed based on preset weights to obtain the temporal domain smoothed image features of the current image to be processed.
[0238] Correspondingly, when the spatial domain smoothed image features of the current image to be processed do not satisfy the similarity condition, step S65 can be replaced by determining the spatial domain smoothed image features of the current image to be processed as the temporal domain smoothed image features of the current image to be processed.
[0239] Optionally, in the aforementioned step S64, a weighted average is performed on the spatial domain smoothed image features of the current image to be processed and the temporal domain smoothed image features of the previous frame of image to be processed. Alternatively, a weighted average is performed on the spatial domain smoothed image features of the current image to be processed and the spatial domain smoothed image features of the previous frame of image to be processed. Correspondingly, the similarity condition in step S63 may be whether the similarity between the spatial domain smoothed image features of the current image to be processed and the temporal domain smoothed image features of the previous frame of image to be processed is less than a preset similarity threshold.
[0240] The above Figure 6 The replacement of the corresponding steps in the method shown is caused by the change in the execution order of the time domain smoothing method and the spatial domain smoothing method when processing the video to be processed, and the change in the processing object when smoothing is performed using the time domain smoothing method. The specific execution process of the replaced steps is the same as Figure 6 The execution process of the corresponding steps in the illustrated embodiment is consistent, and only the processing object needs to be replaced accordingly, which will not be repeated here.
[0241] In other optional embodiments, when the image features of the image to be processed are smoothed using a time domain smoothing method, the step of detecting the similarity as described in step S63 of the aforementioned embodiment and the subsequent step S65 performed when the similarity condition is not met may not be performed, but step S64 may be directly executed, that is, the image features of the current image to be processed and the time domain smoothed image features of the previous frame of the image to be processed are weighted averaged, and the result of the weighted average is used as the time domain smoothed image features of the current image to be processed.
[0242] The effect of executing the step of detecting similarity as described in step S63 is that when the contents of the two frames to be processed in the video to be processed are quite different, or in other words, the contents of the two frames to be processed are not similar, in this case, if the time domain smoothing features of the latter frame to be processed are controlled to be close to the time domain smoothing features of the previous frame to be processed, a large amount of interference will occur in the features of the subsequent processed image generated based on the time domain smoothing features, resulting in the inability of the processed image corresponding to the latter frame to be processed to restore the content of the original image to be processed. Therefore, after detecting that the image features of the current image to be processed meet the similarity condition, the image features of the current image to be processed and the time domain smoothing features of the previous frame to be processed can be weighted averaged. If the similarity condition is not met, the time domain smoothing features of the current image to be processed can be reset to make the time domain smoothing features of the current image to be processed consistent with the image features of the current image to be processed, thereby avoiding the problem that the content of the finally generated processed image is too different from the content of the original image to be processed.
[0243] When smoothing the image features of the image to be processed (or the spatial domain smoothed image features) by using the time domain smoothing method, the weighted average of the image features of the current image to be processed (or the spatial domain smoothed image features) and the time domain smoothed image features of the previous frame of the image to be processed in step S64 is replaced by the weighted average of the image features of the current image to be processed (or the spatial domain smoothed image features) and the image features of the previous frame of the image to be processed (or the spatial domain smoothed image features). The effect is:
[0244] In the original processing method, that is, weighted averaging the image features of the current image to be processed (or spatially smoothed image features) and the time-domain smoothed image features of the previous frame of the image to be processed, it means that in the process of processing the video to be processed, in order to use the time-domain smoothing method to smooth the image features of the next frame of the image to be processed (or spatially smoothed image features), it is necessary to wait for the time-domain smoothing method executed on the image features of the previous frame of the image to be processed (or spatially smoothed image features) to be completed, and obtain the time-domain smoothed image features of the previous frame of the image to be processed. As a result, when the time-domain smoothing method is used to process the entire video to be processed, only the image features of one frame of the image to be processed in the video to be processed (or spatially smoothed image features) can be processed at a time, and the processing speed is slow.
[0245] After the replacement, the image features of any frame of the image to be processed in the video to be processed (or the spatial domain smoothed image features) are smoothed by the time domain smoothing method. There is no need to rely on the processing result of the previous frame of the image to be processed after being processed by the time domain smoothing method, that is, there is no need to wait for the time domain smoothed image features of the previous frame of the image to be processed. In this way, if there are sufficient computing resources, the image features of multiple frames of the image to be processed in the video to be processed (or the spatial domain smoothed image features) can be smoothed by the time domain smoothing method at the same time, thereby improving the speed of processing the video to be processed by the time domain smoothing method.
[0246] It should be noted that in the method provided in the embodiment of the present disclosure, when executing the time domain smoothing method, there is no limitation on whether to perform a weighted average of the spatial domain smoothed image features of the current image to be processed and the time domain smoothed image features of the previous frame of image to be processed, or to perform a weighted average of the spatial domain smoothed image features of the current image to be processed and the spatial domain smoothed image features of the previous frame of image to be processed.
[0247] On the one hand, when starting to process the video to be processed, you can arbitrarily specify whether to perform the time domain smoothing method in the former weighted average method or the latter weighted average method during this processing. If the former weighted average method is specified, when processing each frame of the video to be processed, the spatial domain smoothed image features of the current image to be processed and the temporal domain smoothed image features of the previous frame to be processed are weighted averaged. If the latter weighted average method is specified, when processing each frame of the video to be processed, the spatial domain smoothed image features of the current image to be processed and the spatial domain smoothed image features of the previous frame to be processed are weighted averaged.
[0248] On the other hand, the weighted averaging method may not be specified in advance. In this case, during the video processing process, if the previous frame of the image to be processed of the current frame has both image features that have not been time-smoothed (or spatially smoothed image features) and time-smoothed image features that have been time-smoothed, then any one of these two image features can be selected to perform weighted averaging on the image features (or spatially smoothed image features) of the image to be processed of the current frame. Further, optionally, when weighted averaging the elements of the image features (or spatially smoothed image features) of the image to be processed of the current frame one by one, for some of the elements, the elements of the image features (or spatially smoothed image features) of the previous frame that have not been time-smoothed are used for weighted averaging, and for other elements, the elements of the time-smoothed image features of the previous frame that have been time-smoothed are used for weighted averaging.
[0249] Compared with the latter weighting method mentioned above, and the scheme of arbitrarily changing the weighting method during the processing process, specifying to uniformly execute the time domain smoothing method according to the former weighting method during the entire video processing process can obtain a better time domain smoothing effect. Specifically, the time domain smoothing method is essentially to make the pixels in the image after the next frame is processed approach the pixels at the corresponding position in the image after the previous frame is processed by weighted averaging, thereby reducing the flickering phenomenon in the time domain. Uniformly executing the time domain smoothing method in the former weighted averaging method can make the elements of the time domain smoothing image features of the next frame closer to the elements at the corresponding position of the time domain smoothing image features of the previous frame, so that the pixels of the image after the next frame is processed are closer to the elements at the corresponding position in the image after the previous frame is processed, thereby obtaining a better time domain smoothing effect.
[0250] Finally, the image processing method provided by the present disclosure is described by taking the processing process of a video to be processed as an example. After determining a video to be processed, each frame of the video to be processed is equivalent to a frame of image to be processed. First, each frame of the image to be processed of the video to be processed can be input into the image processing network to obtain the image features of each frame of the image to be processed.
[0251] Then, you can first use Figure 6 The temporal smoothing method shown in the figure performs smoothing on the image features of each frame of the image to be processed, thereby obtaining the temporal smoothed image features of each frame of the image to be processed. Figure 6 , which will not be described in detail here.
[0252] After obtaining the time domain smoothed image features of each frame of the image to be processed, we can use Figure 5 The spatial domain smoothing method shown smoothes the time domain smoothed image features of each frame of the image to be processed to obtain the spatial domain smoothed image features of each frame of the image to be processed, and uses the spatial domain smoothed image features of each frame of the image to be processed as the smoothed image features of this frame of the image to be processed.
[0253] Finally, for each frame of the image to be processed, the smooth image features of this frame of the image to be processed are used to generate a processed image corresponding to this frame of the image to be processed. Each frame of the processed image is combined according to the order of the corresponding image to be processed in the video to be processed, and the processed video corresponding to the video to be processed can be obtained.
[0254] Depending on the specific image processing task, the specific execution process of generating a processed image corresponding to the image to be processed according to the smoothed image features corresponding to the image to be processed is also different for any embodiment of the present disclosure.
[0255] In some image processing tasks, it is necessary to use smooth image features to adjust the original image to be processed in order to obtain the processed image.
[0256] For example, image enhancement is such an image processing task. Image enhancement mainly adjusts the pixel values of each pixel in the image to be processed to purposefully enhance the image's features in certain aspects, thereby improving the visual effect.
[0257] When the image processing method provided by the present disclosure is used to process an image enhancement task, the specific execution process of generating a processed image corresponding to the image to be processed according to the smoothed image features corresponding to the image to be processed may be:
[0258] Each feature map contained in the smoothed image feature of the image to be processed is accumulated to obtain a smoothed feature map (equivalent to a matrix) corresponding to the image to be processed.
[0259] Then, coefficient interpolation is performed on the smooth feature map to increase the size of the smooth feature map so that the interpolated image enhancement features (which can also be understood as a matrix) are consistent with the size of the image to be processed. Coefficient interpolation is an existing method for increasing the resolution of an image. In the present disclosure, its basic working principle can be that for any two elements in the smooth feature map that are located in the same row or in the same column, the two elements are substituted into a preset interpolation formula, and then the calculation result output by the interpolation formula is used as a new element between the two elements, thereby completing a coefficient interpolation of the two elements. When the coefficient interpolation is performed on the entire smooth feature map to obtain image enhancement features of the same size as the image to be processed, the coefficient interpolation can be repeated multiple times for a part of the elements in the smooth feature map (for example, elements of a specified number of rows or columns) according to the size difference between the smooth feature map and the image to be processed, until the image enhancement features of the same size as the image to be processed are obtained.
[0260] For example, coefficients of every two elements in the same column of the first row and the second row are interpolated, so that a new row of elements is added between the original first row and the second row, and then the above process can be repeated between the original first row and the newly added second row.
[0261] Since the size of the image enhancement feature is consistent with the image to be processed, each element of the image enhancement feature corresponds to a pixel at the same position in the image to be processed. Therefore, for each pixel of the image to be processed, the pixel value of the pixel can be adjusted using the element corresponding to the pixel in the image enhancement feature to obtain the adjusted pixel. After all the pixels are adjusted, all the adjusted pixels constitute the processed image.
[0262] Optionally, the above adjustment process may be that, for a pixel point of the image to be processed, the pixel value of the pixel point is multiplied by the corresponding element in the image enhancement feature, and the obtained product is used as the adjusted pixel value of the pixel point, thereby obtaining the adjusted pixel point.
[0263] In other image enhancement methods, the original pixel value and the corresponding element may also be added or divided, and the present disclosure does not limit this.
[0264] In another part of the image processing task, the smoothed image features can be directly merged into a processed low-resolution image, and then the coefficients of the processed low-resolution image can be interpolated to obtain a processed image with the same resolution as the image to be processed.
[0265] For example, stylization of an image is a task of this type. Image stylization refers to obtaining a stylized image with a specific artistic effect by adjusting the pixel values of each pixel in the image. When the method provided by the present disclosure is used to perform an image stylization task, the smoothed image features can be directly summed to obtain a low-resolution stylized image, and then the coefficients of the low-resolution stylized image can be interpolated to obtain a stylized image with the same resolution as the image to be processed.
[0266] Another exemplary embodiment of the present disclosure further provides an image processing method, please refer to Figure 7 , the method may include the following steps:
[0267] In step S71, the image features of each image to be processed are extracted using an image processing network.
[0268] The specific execution process of step S71 can be referred to Figure 4 The corresponding steps S41 and S42 in the embodiment.
[0269] In step S72, for each image to be processed, a processed image is generated according to the image features of the image to be processed.
[0270] The specific process of step S72 may refer to step 44.
[0271] In step S73, each of the once-processed images is displayed to the user, and feedback from the user on any one or more of the once-processed images is obtained.
[0272] If the multiple images to be processed are independent static images, then in step S73, the images processed once are displayed to the user by determining an interval time, and for each image processed once, the image processed once is continuously displayed on the display device until the duration reaches the set interval time, and then the next image processed once is switched to be displayed. For example, the interval time can be set to 60 seconds, and then each image processed once is continuously displayed for 60 seconds, and then the next image processed once is switched to be displayed for another 60 seconds.
[0273] If the aforementioned multiple images to be processed are multiple consecutive frames of images to be processed in a video to be processed, then the processed images are displayed to the user in step S73, which may be to combine all the images to be processed into a processed video according to the order of the corresponding frames in the video to be processed, and play the processed video at a lower video playback speed. For example, if the video to be processed can be played for 20 seconds at a normal speed, the processed video can be played at one-fourth of the normal speed, that is, the processed video will be played for 80 seconds.
[0274] In step S74, smoothing is performed on the image features of each abnormal once-processed image to obtain corresponding smoothed image features.
[0275] Whether any image processed once has an abnormality can be determined based on user feedback.
[0276] For example, when the images processed once are multiple independent static images, the user can use a variety of methods including drawing rectangular boxes to mark the areas with unnatural transitions in any of the images processed once. If the user marks a certain image processed once, then this image processed once is determined to be an abnormal image processed once.
[0277] When multiple once-processed images are composed of a once-processed video and played, the user can pause the video playback when observing unnatural transitions or flickers, and then use various methods including drawing rectangular boxes to mark the areas with unnatural transitions or flickers in the currently displayed once-processed image in the paused screen.
[0278] Furthermore, if the user marks the area where the above-mentioned abnormality exists (i.e., there is unnatural transition or flicker) in the image after a single processing where the abnormality exists, then when executing step S74, a part of the image features that affect the area where the abnormality exists in the image after a single processing can be determined based on the area marked by the user. When the corresponding image features are smoothed using the spatial domain smoothing method and the temporal domain smoothing method, only the image features that affect the area where the abnormality exists in the image after a single processing are smoothed, and other image features are not smoothed.
[0279] In step S75, for each smoothed image feature of the image to be processed, a secondary processed image is generated according to the smoothed image feature.
[0280] The specific execution process of step S75 is similar to the aforementioned step S34 and will not be described in detail.
[0281] For the aforementioned step S74, if the user marks a frame of the once-processed image as abnormal in the once-processed video, then when smoothing the image features, firstly detect whether the image features of the once-processed image with the abnormality meet the similarity condition. If so, then starting from the once-processed image, detect one by one whether the image features of each frame of the image to be processed before the once-processed image with the abnormality meet the similarity condition until a frame of image that is located before the once-processed image with the abnormality and does not meet the similarity condition is detected, and then use this frame of the image to be processed that does not meet the similarity condition as the aforementioned Figure 6 The first frame of the image to be processed in the corresponding embodiment is used to perform the following steps: Figure 6 The method shown is to process the image features of each frame of the image to be processed using the time domain smoothing method.
[0282] On the one hand, when there are multiple images to be processed, it is possible that only the image features corresponding to a part of the images to be processed will cause the corresponding processed images to have poor continuity in the time domain and spatial domain, while the image features of other images to be processed may not have such problems. The present disclosure avoids processing image features that will not cause poor continuity in the time domain and spatial domain by first obtaining a processed image and determining the existence of an abnormal processed image based on user feedback, thereby reducing the computing resources consumed by the image processing method provided by the present disclosure.
[0283] Furthermore, on the premise that the user marks the abnormal areas in the image after a processing, this embodiment can further identify a part of the image features that affect the abnormal areas, and only smooth these part of the image features, further reducing the computing resources consumed when processing the image.
[0284] In order to intuitively understand the beneficial effects of the method provided by the present disclosure, Figure 8 A schematic diagram showing a comparison between an image processed by the image processing method provided by the present disclosure and an image before processing is shown. Fig. 9 A schematic diagram of two consecutive processed images in a video obtained based on the image processing method provided by the present disclosure is shown.
[0285] Please refer to Figure 8 , and combined with Figure 1 , Figure 8 and Figure 1 All of them are stylized processing of the images before processing. It can be found that the images obtained by the method provided by the present disclosure are Figure 8 In the processed image on the right, the background parts in the two rectangular frames do not appear as shown in the figure. Figure 1 The unnatural transition in the image is not obvious, but is similar to the image before processing, that is, the background part is basically the same.
[0286] Please refer to Fig. 9 , and combined with Figure 3 , Fig. 9 and Figure 3 are two consecutive processed images obtained by stylizing the previous and next frame images in the video. It can be found that compared with Figure 3 , Fig. 9 In the rectangular frames of the two processed images, the bright spots have disappeared.
[0287] In combination with the above-mentioned figures, it can be found that the image processing method provided by the present disclosure can effectively improve the continuity of the image processed by the image processing network in the time domain and the spatial domain.
[0288] Fig.10 is a block diagram of a device for processing an image according to an exemplary embodiment. The device may include:
[0289] An acquisition unit 1001 is configured to acquire an image to be processed;
[0290] The processing unit 1002 is configured to input the image to be processed into the image processing network to obtain image features of the image to be processed;
[0291] The first smoothing unit 1003 is configured to perform smoothing processing on each element of the image feature of the image to be processed and its adjacent elements to obtain a spatial domain smoothed image feature of the image to be processed.
[0292] The adjacent elements of an element include: elements located in a preset range centered on the element in the image features of the image to be processed.
[0293] The generating unit 1004 is configured to generate a processed image corresponding to the image to be processed according to the spatial domain smoothed image features of the image to be processed.
[0294] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0295] The first smoothing unit performs smoothing processing on the image features so that the elements in the obtained spatial domain smoothed image features are close to the adjacent elements, thereby improving the continuity of the processed image generated according to the spatial domain smoothed image features in the spatial domain.
[0296] Optionally, the first smoothing unit 1003 includes:
[0297] The determination unit is configured to determine the size of the smoothing kernel of the feature map according to the attribute of the feature map contained in the image feature.
[0298] The updating unit is configured to align each element of each feature map of the image feature with the center of the smoothing kernel, and update each element to the arithmetic mean or weighted mean of all elements covered by the smoothing kernel.
[0299] The obtaining unit is configured to use the set of all updated feature maps as the spatial domain smoothed image features of the image to be processed.
[0300] Optionally, the determining unit may include:
[0301] A first determining subunit is configured to determine the size of the smoothing kernel of the feature map according to a comparison result of the size of the feature map, a preset maximum smoothing kernel size, and a preset minimum smoothing kernel size;
[0302] Alternatively, the second determination subunit is configured to calculate the gradient value of the feature map, and obtain the size of the smoothing kernel of the feature map based on the gradient value of the feature map.
[0303] Optionally, generate a unit, including:
[0304] The generating subunit is configured to sum all feature maps contained in the spatial domain smooth image feature of the image to be processed to obtain a smooth feature map, and to perform coefficient interpolation on the smooth feature map to obtain an image enhancement feature of the image to be processed; wherein the size of the image enhancement feature of the image to be processed is consistent with the size of the image to be processed;
[0305] The adjustment unit is configured to adjust the pixel value of each pixel of the image to be processed according to the element corresponding to the pixel in the image enhancement feature of the image to be processed, obtain the adjusted pixel, and combine all the adjusted pixel into a processed image.
[0306] Fig.11is a block diagram of a device for processing an image according to an exemplary embodiment, and the device may include:
[0307] The acquisition unit 1101 is configured to acquire an image to be processed.
[0308] The processing unit 1102 is configured to input the image to be processed into the image processing network to obtain image features of the image to be processed.
[0309] The second smoothing unit 1103 is configured to perform smoothing processing on each element of the image feature of the image to be processed and the corresponding element of the element to obtain the time-domain smoothed image feature of the image to be processed.
[0310] The corresponding element of the element includes: the element at the same position as the element in the time domain smoothed image feature of the previous image adjacent to the image to be processed in the time domain, or the element at the same position as the element in the image feature of the previous image adjacent to the image to be processed in the time domain.
[0311] The first smoothing unit 1104 is configured to perform smoothing processing on each element of the time-domain smoothed image feature of the image to be processed and its adjacent elements to obtain the space-domain smoothed image feature of the image to be processed.
[0312] The adjacent elements of an element include: elements located in a preset range centered on the element in the image features of the image to be processed.
[0313] The generating unit 1105 is configured to generate a processed image corresponding to the image to be processed according to the spatial domain smoothed image features of the image to be processed.
[0314] Optionally, the second smoothing unit includes:
[0315] The first weighted averaging unit is configured to perform weighted averaging on the image feature of the image to be processed and the time-domain smoothed image feature of the previous image or the image feature of the previous image to obtain the time-domain smoothed image feature of the image to be processed.
[0316] Optionally, the second smoothing unit further includes:
[0317] A first control unit is configured to control a first weighted averaging unit to perform weighted averaging of the image feature of the image to be processed, the time-domain smoothed image feature of the previous image or the image feature of the previous image in response to the image feature of the image to be processed having a similarity with the time-domain smoothed image feature of the previous image or the image feature of the previous image being not less than a similarity threshold, so as to obtain the time-domain smoothed image feature of the image to be processed;
[0318] It is also configured to respond to the image features of the image to be processed, and the similarity with the time-domain smoothed image features or image features of the previous image is less than a similarity threshold, and use the image features of the image to be processed as the time-domain smoothed image features of the image to be processed.
[0319] Fig.12 is a block diagram of a device for processing an image according to an exemplary embodiment, and the device may include:
[0320] The acquisition unit 1201 is configured to acquire an image to be processed.
[0321] The processing unit 1202 is configured to input the image to be processed into the image processing network to obtain image features of the image to be processed.
[0322] The first smoothing unit 1203 is configured to perform smoothing processing on each element of the time-domain smoothed image feature of the image to be processed and its adjacent elements to obtain the space-domain smoothed image feature of the image to be processed.
[0323] The adjacent elements of an element include: elements located in a preset range centered on the element in the image features of the image to be processed.
[0324] The third smoothing unit 1204 is configured to perform smoothing processing on each element of the spatial domain smoothed image feature of the image to be processed and the corresponding element of the element to obtain the temporal domain smoothed image feature of the image to be processed.
[0325] The corresponding element of the element includes: the element at the same position as the element in the time domain smoothed image feature of the previous image adjacent to the image to be processed in the time domain, or the element at the same position as the element in the spatial domain smoothed image feature of the previous image adjacent to the image to be processed in the time domain.
[0326] The generating unit 1205 generates a processed image corresponding to the image to be processed according to the time domain smoothed image feature of the image to be processed.
[0327] Optionally, the third smoothing unit includes:
[0328] The second weighted averaging unit is configured to perform weighted averaging on the spatial domain smoothed image features of the image to be processed and the temporal domain smoothed image features of the previous image or the spatial domain smoothed image features of the previous image to obtain the temporal domain smoothed image features of the image to be processed.
[0329] Optionally, the third smoothing unit further includes:
[0330] The second control unit is configured to control the second weighted averaging unit to perform weighted averaging of the spatial domain smoothed image feature of the image to be processed and the temporal domain smoothed image feature of the previous image or the spatial domain smoothed image feature of the previous image in response to the similarity between the spatial domain smoothed image feature of the image to be processed and the temporal domain smoothed image feature of the previous image or the spatial domain smoothed image feature of the previous image, so as to obtain the temporal domain smoothed image feature of the image to be processed;
[0331] It is also configured to respond to the spatial domain smooth image features of the image to be processed, and the similarity between the spatial domain smooth image features of the image to be processed and the temporal domain smooth image features or spatial domain smooth image features of the previous image is less than a similarity threshold, and use the spatial domain smooth image features of the image to be processed as the temporal domain smooth image features of the image to be processed.
[0332] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0333] Fig.13 It is a block diagram of an electronic device for executing the aforementioned image processing method according to an exemplary embodiment. The electronic device includes a processor 1301 and a memory 1302 for storing instructions executable by the processor.
[0334] The processor 1301 is configured to execute the instructions to implement the image processing method provided by any embodiment of the present disclosure.
[0335] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, wherein the instructions may be Fig.13 The processor 1301 of the electronic device shown executes to complete the above method. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0336] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed, it is used to implement the image processing method provided by any embodiment of the present disclosure.
[0337] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0338] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. An image processing method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into an image processing network to obtain image features of the image to be processed; Performing smoothing processing on each element of the image feature of the image to be processed and the adjacent elements of the element to obtain the spatial domain smoothed image feature of the image to be processed; wherein: the adjacent elements of the element include: elements located in a preset range centered on the element in the image feature of the image to be processed; Generating a processed image corresponding to the image to be processed according to the spatial domain smoothed image features of the image to be processed; Wherein, after inputting the image to be processed into an image processing network to obtain the image features of the image to be processed, the method further includes: Performing smoothing processing on each element of the image feature of the image to be processed and the corresponding element of the element to obtain the time-domain smoothed image feature of the image to be processed; the corresponding element of the element includes: the element at the same position as the element in the time-domain smoothed image feature of a previous image adjacent to the image to be processed in the time domain, or the element at the same position as the element in the image feature of a previous image adjacent to the image to be processed in the time domain; Wherein: the step of smoothing each element of the image feature of the image to be processed and its adjacent elements to obtain the spatial domain smoothed image feature of the image to be processed includes: smoothing each element of the temporal domain smoothed image feature of the image to be processed and its adjacent elements to obtain the spatial domain smoothed image feature of the image to be processed.
2. The image processing method according to claim 1, characterized in that: After smoothing each element of the image feature of the image to be processed and its adjacent elements to obtain the spatial domain smoothed image feature of the image to be processed, the method further includes: Smoothing each element of the spatial domain smoothed image feature of the image to be processed and the corresponding element of the element to obtain the temporal domain smoothed image feature of the image to be processed; the corresponding element of the element includes: the element at the same position as the element in the temporal domain smoothed image feature of a previous image adjacent to the image to be processed in the temporal domain, or the element at the same position as the element in the spatial domain smoothed image feature of a previous image adjacent to the image to be processed in the temporal domain; Wherein: generating a processed image corresponding to the image to be processed according to the spatial domain smoothed image features of the image to be processed includes: generating a processed image corresponding to the image to be processed according to the temporal domain smoothed image features of the image to be processed.
3. The image processing method according to claim 1, characterized in that: The step of performing smoothing on each element of the image feature of the image to be processed and its adjacent elements to obtain a spatial domain smoothed image feature of the image to be processed includes: Determining the size of the smoothing kernel of the feature map according to the attribute of the feature map contained in the image feature; Aligning each element of each feature map of the image feature with the center of the smoothing kernel, and updating each of the elements to the arithmetic mean or weighted mean of all elements covered by the smoothing kernel; The set of all updated feature maps is used as the spatial domain smoothed image feature of the image to be processed.
4. The image processing method according to claim 3, characterized in that: The step of determining the size of the smoothing kernel of the feature map according to the attribute of the feature map contained in the image feature includes: Determining the size of the smoothing kernel of the feature map according to a comparison result of the size of the feature map, a preset maximum smoothing kernel size, and a preset minimum smoothing kernel size; Alternatively, the gradient value of the feature map is calculated, and the size of the smoothing kernel of the feature map is calculated according to the gradient value of the feature map.
5. The image processing method according to claim 1, characterized in that: The step of smoothing each element of the image feature of the image to be processed and the corresponding element of the element to obtain the time-domain smoothed image feature of the image to be processed includes: The image features of the image to be processed and the time-domain smoothed image features of the previous image or the image features of the previous image are weighted averaged to obtain the time-domain smoothed image features of the image to be processed.
6. The image processing method according to claim 2, characterized in that: The step of smoothing each element of the spatial domain smoothed image feature of the image to be processed and its corresponding element to obtain the temporal domain smoothed image feature of the image to be processed includes: The spatial domain smoothed image features of the image to be processed and the temporal domain smoothed image features of the previous image or the spatial domain smoothed image features of the previous image are weighted averaged to obtain the temporal domain smoothed image features of the image to be processed.
7. The image processing method according to claim 5, characterized in that: Also includes: In response to the image feature of the image to be processed having a similarity with the time-domain smoothed image feature or the image feature of the previous image being not less than a similarity threshold, performing weighted averaging of the image feature of the image to be processed and the time-domain smoothed image feature of the previous image or the image feature of the previous image to obtain the time-domain smoothed image feature of the image to be processed; In response to the image feature of the image to be processed having a similarity with the time-domain smoothed image feature or the image feature of the previous image being less than a similarity threshold, the image feature of the image to be processed is used as the time-domain smoothed image feature of the image to be processed.
8. The image processing method according to claim 6, characterized in that: Also includes: In response to the spatial domain smoothed image feature of the image to be processed having a similarity with the temporal domain smoothed image feature or the spatial domain smoothed image feature of the previous image being not less than a similarity threshold, weighted averaging the spatial domain smoothed image feature of the image to be processed and the temporal domain smoothed image feature of the previous image or the spatial domain smoothed image feature of the previous image is performed to obtain the temporal domain smoothed image feature of the image to be processed; In response to the spatial domain smooth image features of the image to be processed, the similarity between the spatial domain smooth image features and the temporal domain smooth image features or the spatial domain smooth image features of the previous image is less than a similarity threshold, and the spatial domain smooth image features of the image to be processed are used as the temporal domain smooth image features of the image to be processed.
9. The image processing method according to claim 1, characterized in that: The step of generating a processed image corresponding to the image to be processed according to the spatial domain smoothed image features of the image to be processed comprises: Sum all feature maps contained in the spatial domain smooth image feature of the image to be processed to obtain a smooth feature map, and perform coefficient interpolation on the smooth feature map to obtain the image enhancement feature of the image to be processed; wherein the size of the image enhancement feature of the image to be processed is consistent with the size of the image to be processed; For each pixel of the image to be processed, the pixel value of the pixel is adjusted according to the element corresponding to the pixel in the image enhancement feature of the image to be processed to obtain the adjusted pixel, and all the adjusted pixel points are combined into a processed image.
10. An image processing device, characterized in that: include: An acquisition unit, configured to acquire an image to be processed; A processing unit, configured to input the image to be processed into an image processing network to obtain image features of the image to be processed; A first smoothing unit is configured to perform smoothing processing on each element of the image feature of the image to be processed and its adjacent elements to obtain a spatial domain smoothed image feature of the image to be processed; The adjacent elements of the element include: elements located within a preset range centered on the element in the image features of the image to be processed; A generating unit configured to generate a processed image corresponding to the image to be processed according to the spatial domain smoothed image features of the image to be processed; Among them, it also includes: A second smoothing unit is used to perform smoothing processing on each element of the image feature of the image to be processed and the corresponding element of the element to obtain the time-domain smoothed image feature of the image to be processed; the corresponding element of the element includes: the element at the same position as the element in the time-domain smoothed image feature of a previous image adjacent to the image to be processed in the time domain, or the element at the same position as the element in the image feature of the previous image adjacent to the image to be processed in the time domain; Wherein: when the first smoothing unit performs smoothing processing on each element of the image feature of the image to be processed and the adjacent elements of the element to obtain the spatial domain smoothed image feature of the image to be processed, it is used to: perform smoothing processing on each element of the time domain smoothed image feature of the image to be processed and the adjacent elements of the element to obtain the spatial domain smoothed image feature of the image to be processed.
11. The image processing device according to claim 10, characterized in that: Also includes: A third smoothing unit is used to perform smoothing processing on each element of the spatial domain smoothed image feature of the image to be processed and the corresponding element of the element to obtain the temporal domain smoothed image feature of the image to be processed; The corresponding element of the element includes: an element at the same position as the element in the time domain smoothed image feature of a previous image adjacent to the image to be processed in the time domain, or an element at the same position as the element in the space domain smoothed image feature of a previous image adjacent to the image to be processed in the time domain; Wherein: when the generating unit generates a processed image corresponding to the image to be processed according to the spatial domain smoothed image features of the image to be processed, it is used to: generate a processed image corresponding to the image to be processed according to the temporal domain smoothed image features of the image to be processed.
12. The image processing device according to claim 10, characterized in that: The first smoothing unit comprises: A determination unit, configured to determine a size of a smoothing kernel of the feature map according to an attribute of the feature map contained in the image feature; An updating unit, used for aligning each element of each feature map of the image feature with the center of the smoothing kernel, and updating each of the elements to the arithmetic mean or weighted mean of all elements covered by the smoothing kernel; The obtaining unit is used to use the set of all updated feature maps as the spatial domain smoothed image features of the image to be processed.
13. The image processing device according to claim 12, characterized in that: The determining unit comprises: A first determining subunit, configured to determine the size of the smoothing kernel of the feature map according to a comparison result of the size of the feature map, a preset maximum smoothing kernel size, and a preset minimum smoothing kernel size; Alternatively, the second determination subunit is used to calculate the gradient value of the feature map, and obtain the size of the smoothing kernel of the feature map according to the gradient value of the feature map.
14. The image processing device according to claim 10, characterized in that: The second smoothing unit comprises: The first weighted averaging unit is used to perform weighted averaging on the image features of the image to be processed and the time-domain smoothed image features of the previous image or the image features of the previous image to obtain the time-domain smoothed image features of the image to be processed.
15. The image processing device according to claim 11, characterized in that: The third smoothing unit comprises: The second weighted averaging unit is used to perform weighted averaging on the spatial domain smoothed image features of the image to be processed and the temporal domain smoothed image features of the previous image or the spatial domain smoothed image features of the previous image to obtain the temporal domain smoothed image features of the image to be processed.
16. The image processing device according to claim 14, characterized in that: Also includes: A first control unit, configured to control the first weighted averaging unit to perform weighted averaging of the image feature of the image to be processed and the time-domain smoothed image feature of the previous image or the image feature of the previous image in response to the image feature of the image to be processed having a similarity with the time-domain smoothed image feature of the previous image or the image feature of the previous image being not less than a similarity threshold, so as to obtain the time-domain smoothed image feature of the image to be processed; It is also used to respond to the image feature of the image to be processed, and the similarity with the time-domain smoothed image feature or image feature of the previous image is less than a similarity threshold, and use the image feature of the image to be processed as the time-domain smoothed image feature of the image to be processed.
17. The image processing device according to claim 15, characterized in that: Also includes: a second control unit, configured to control the second weighted averaging unit to perform weighted averaging of the spatial domain smoothed image feature of the image to be processed and the temporal domain smoothed image feature of the previous image or the spatial domain smoothed image feature of the previous image, in response to the similarity between the spatial domain smoothed image feature of the image to be processed and the temporal domain smoothed image feature of the previous image or the spatial domain smoothed image feature of the previous image being not less than a similarity threshold, so as to obtain the temporal domain smoothed image feature of the image to be processed; It is also used to respond to the spatial domain smooth image features of the image to be processed, and the similarity between the spatial domain smooth image features and the temporal domain smooth image features or the spatial domain smooth image features of the previous image is less than a similarity threshold, and use the spatial domain smooth image features of the image to be processed as the temporal domain smooth image features of the image to be processed.
18. The image processing device according to claim 10, characterized in that: The generating unit comprises: A generating subunit is used to sum all feature maps contained in the spatial domain smooth image feature of the image to be processed to obtain a smooth feature map, and to perform coefficient interpolation on the smooth feature map to obtain the image enhancement feature of the image to be processed; wherein the size of the image enhancement feature of the image to be processed is consistent with the size of the image to be processed; The adjustment unit is used to adjust the pixel value of each pixel of the image to be processed according to the element corresponding to the pixel in the image enhancement feature of the image to be processed, obtain the adjusted pixel, and combine all the adjusted pixel into a processed image.
19. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 9.
20. A computer storage medium, characterized in that When the instructions in the computer storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method according to any one of claims 1 to 9.
