Image processing method based on deep learning and Fourier time-space domain transformation

Abstract

The invention discloses an image processing method based on deep learning and Fourier time-space domain transformation, and the method comprises the specific steps: image processing: collecting an original image frame from a video, inputting the original image frame into a convolutional neural network, carrying out the convolution of horizontal and vertical latitudes on an image, outputting an operation result through a convolution kernel, and carrying out the image processing; a modified image is obtained; and carrying out feature extraction through BN operator operation: carrying out normalization processing on a result output by the deep convolution operation, unifying data distribution of different network layers, carrying out data transformation and reconstruction after normalization, and recovering feature information of original data. The BN operator operation is carried out on the result output by the deep convolution operation, the BN operator operation enables the distribution of input data of each layer in the network to be relatively stable, the learning speed of the model is increased, the regularization effect on the model is achieved to a certain extent, and the analyzed result is more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image processing method based on deep learning and Fourier space-time domain transformation. BACKGROUND

[0002] Traditional image optical flow method has some obvious shortcomings in judging the movement of objects, the most important of which is that it can only be used in static background conditions. Since the background will change, the traditional image optical flow method will be affected, resulting in inaccurate analysis results. In addition, the traditional image optical flow method also has the problems of high time complexity and large amount of calculation, which will consume a large amount of computing resources, making real-time application unfeasible.

[0003] In this traditional method, four functions are used, including Gaussian kernel function, spatial gradient, temporal gradient and error function.

[0004] The Gaussian kernel function is a smoothing filter used for Gaussian blur processing of images. The problems faced by the Gaussian kernel in the image field include: scale selection problem: the standard deviation of the Gaussian kernel determines its angular frequency, so different standard deviations will lead to different scale selection, how to choose the most appropriate scale is a problem; edge response problem: Gaussian kernel smoothing image will cause the loss of information of image edge, thus reducing the sensitivity of edge response, which will lead to detection failure at the edge; calculation complexity problem: the calculation amount of Gaussian kernel is large, especially for large kernel and high-dimensional image, the calculation complexity will increase sharply, resulting in low efficiency.

[0005] The spatial gradient is used to calculate the rate of change of the image in space, which can be calculated by Sobel operator or Prewitt operator. The problems faced by Sobel operation in the image field include: noise sensitivity problem: Sobel operation is sensitive to noise, noise will cause inaccurate gradient value, thus affecting the extracted features. Directionality problem: Sobel operation can only detect horizontal and vertical edges of the image, and is not accurate enough for detecting diagonal edges. Threshold selection problem: the gradient map output by Sobel operation needs to be thresholded, but how to choose the most appropriate threshold is a problem.

[0006] The Prewitt operator is a discrete differential operator used to calculate the gradient of an image. The disadvantages of the Prewitt operator include: for images with low edge intensity, the Prewitt operator may fail to detect. For images with non-obvious edge direction, the Prewitt operator may detect inaccurate edge direction. The calculation amount of the Prewitt operator is large, especially for large images, the calculation complexity will increase sharply, resulting in low efficiency.

[0007] Temporal gradient: Temporal gradient is used to calculate the rate of change of images over time, which can be calculated by first-order difference operator or second-order difference operator. The problems faced by temporal gradient operation include: brightness change problem: temporal gradient operation is more sensitive to light conditions, for images with large brightness change, temporal gradient may fail. Motion blur problem: for images with serious motion blur, temporal gradient may lead to inaccurate results. Noise sensitivity problem: temporal gradient operation is sensitive to noise, noise will lead to inaccurate gradient value, which will affect feature extraction.

[0008] Error function is used to calculate the error of optical flow vector, so as to minimize the error function to obtain the optimal solution. The most commonly used error function is mean square error function, and the problems faced by error function include: for images with poor light conditions, error function may fail, resulting in inaccurate results. Error function needs efficient solving algorithm, otherwise it will lead to too large calculation amount, so that real-time application becomes infeasible. Error function is sensitive to the selection of initial optical flow vector, so it is necessary to select appropriate initial value to ensure the accuracy of the results.

[0009] Therefore, in order to improve the operation accuracy, an image processing method based on deep learning and Fourier space-time domain transformation is proposed to solve the above problems. SUMMARY

[0010] The purpose of the present application is to provide an image processing method based on deep learning and Fourier space-time domain transformation, and the specific steps of the image processing method include: Step 1: image processing: collecting original image frames from video, inputting convolutional neural network, performing convolution on images in horizontal and vertical latitudes, outputting operation results from convolution kernel, and obtaining a modified image; The specific steps of step 1 are as follows: Step 11: input image is used as information bucket 1 in convolutional neural network, information bucket 1 is composed of three matrices, namely RGB three channels, and each element is an integer between 0 and 255; Step 12: information bucket 2 in convolutional neural network is convolution kernel, that is, a single floating point matrix; Step 13: mix the information of information bucket 1 and information bucket 2, and optimize the convolution result through Fourier transformation The calculation formula is: ; Wherein, And are two continuous functions participating in convolution operation, which are input functions of convolution; is the independent variable of convolution result ; is the integral variable, which represents the function Parameters for translation and inversion; integral sign Indicates integration of a variable over the entire real domain Performing integration operation Indicates Fourier transform of a function Performing Fourier transform to convert time domain function to frequency domain Indicates Fourier transform of a function Performing Fourier transform Indicates inverse Fourier transform to convert frequency domain result back to time domain

[0011] Step 2: Feature extraction by BN operator operation: normalize the output of the deep convolution operation, unify the data distribution of different network layers, and after normalization, transform and reconstruct the data to restore the feature information of the original data, and the specific steps of step 2 are as follows: Step 21: Normalize the convolution data, and the specific formula is as follows: ; ; Where, Indicates the arithmetic mean of the data, Indicates the number of samples, that is, the number of observations contained in the data set; Indicates the specific value of the th sample; Indicates the variance of the data; Indicates the deviation of the th sample value from the mean; Step 22: Introduce linear offset parameters for transformation and reconstruction and , which respectively perform scale and shift operations on the simply normalized data to transform and reconstruct the data and restore the feature information of the original data, and the specific formula is as follows: ; ; Where, is the scaling factor, which performs scale operation on the normalized data to adjust the amplitude of the data; is the offset factor, which performs shift operation on the scaled data to adjust the center position of the data; is the mean of the data; is the variance of the data; is a small positive number to prevent the denominator from being zero; is the simply normalized data; is the transformed and reconstructed data.

[0012] Compared with the prior art, the beneficial effects of the present application are as follows: the present application extracts original image frames from a video, performs convolution in two latitudes of horizontal and vertical on the original image frames, optimizes the convolution data through Fourier transform, performs BN operator operation on the results output by the deep convolution operation, the BN operator operation makes the distribution of the input data in each layer of the network relatively stable, accelerates the model learning speed, to a certain extent, the model is regularized, and the analyzed results are more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a schematic diagram of the steps of the present application; Figure 2 is an effect diagram after step 1 of the present application. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0015] As shown in the accompanying Figure 1 , the present application provides an image processing method based on deep learning and Fourier space-time domain transformation, and the specific steps of the image processing method include: Step 1: image processing: collecting original image frames from a video, inputting a convolutional neural network, performing convolution in two latitudes of horizontal and vertical on the image, and outputting operation results from the convolution kernel, as shown in the accompanying Figure 2 , obtaining a modified image; The specific steps of step 1 are: Step 11: input the image as information bucket 1 in the convolutional neural network, the information bucket 1 is composed of three matrices, i.e. RGB three channels, and each element is an integer between 0 and 255; Step 12: the information bucket 2 in the convolutional neural network is the convolution kernel, i.e. a single floating point matrix; Step 13: mix the information of information bucket 1 and information bucket 2, and optimize the convolution results through Fourier transform , the calculation formula is: ; Among them, and are two continuous functions participating in the convolution operation, which are input functions of the convolution; is the independent variable of the convolution result ; is the integral variable, representing the function Parameters for translation and reversal operations; integral sign Represents the variable over the entire real number field. Perform integration; Represents the function Perform a Fourier transform to convert the time-domain function to the frequency domain; Represents the function Perform a Fourier transform; This indicates the inverse Fourier transform, which converts the frequency domain result back to the time domain.

[0016] Step 2: When training a deep neural network, changes in the parameters of each hidden layer will change the input of the next layer, thus altering the distribution of each batch of training data. This causes the network to need to fit different data distributions in each iteration, increasing training complexity and the risk of overfitting. Therefore, we need to be very careful in setting the learning rate, initializing weights, and other parameter update strategies. To ensure that each batch of data in the network layers is under the same data distribution, we need to add normalization processing before the input of each layer. Specifically, this involves subtracting the mean from all elements in the current batch and then dividing by the standard deviation. This normalization operation adds additional constraints to the original data, thereby enhancing the model's generalization ability. The convolutional data is normalized using the following formula: ; ; in, This represents the arithmetic mean of the data set. This indicates the number of samples, that is, the number of observations contained in the dataset; Indicates the first The specific values ​​that each sample can take; This represents the variance of the data set; Indicates the first The deviation of each sample value from the mean; Since simple normalization forces the data distribution to have a mean of 0 and a standard deviation of 1, it may destroy the characteristics of the original data. In order to restore the original data distribution, the BN operator introduces a linear offset parameter for transformation and reconstruction. and These operations perform scale and shift on the simply normalized data, respectively, to transform and reconstruct the data, restoring the feature information of the original data. The specific formulas are as follows: ; ; in, is a scaling factor, performing a scale operation on the normalized data to adjust the amplitude of the data; is an offset factor, performing a shift operation on the scaled data to adjust the center position of the data; is the mean of the data; is the variance of the data; is a small positive number, used to prevent the denominator from being zero; is the simply normalized data; is the data after transformation and reconstruction.

[0017] The technical solutions of the present application or the technical solutions inspired by the technical solutions of the present application, which achieve the above technical effects, are all within the protection scope of the present application.

Claims

1. An image processing method based on deep learning and Fourier spatiotemporal transform, characterized in that: The specific steps of this image processing method include: Step 1: Image Processing: Acquire raw image frames from the video, input them into a convolutional neural network, perform convolution on the image in both horizontal and vertical dimensions, and output the results from the convolution kernel to obtain a modified image; Step 2: Feature extraction using BN operator: Normalize the output of depthwise convolution to unify the data distribution of different network layers. After normalization, reconstruct the data to restore the feature information of the original data.

2. The image processing method based on deep learning and Fourier spatiotemporal transform according to claim 1, characterized in that: The specific steps of step 1 are as follows: Step 11: Use the input image as the first information bucket in the convolutional neural network. The first information bucket consists of three matrices, namely RGB three channels, where each element is an integer between 0 and 255. Step 12: The second information bucket in the convolutional neural network is the convolution kernel, which is a single floating-point matrix; Step 13: Mix the information from information bucket 1 and information bucket 2, and optimize the convolution result using Fourier transform. The calculation formula is: ； in, and These are two continuous functions participating in the convolution operation, and the input function for the convolution. It is the result of convolution. The independent variable; It is the integration variable, representing the integral of the function. Parameters for translation and reversal operations; integral sign Represents the variable over the entire real number field. Perform integration; Represents the function Perform a Fourier transform to convert the time-domain function to the frequency domain; Represents the function Perform a Fourier transform; This indicates the inverse Fourier transform, which converts the frequency domain result back to the time domain.

3. The image processing method based on deep learning and Fourier spatiotemporal transform according to claim 1, characterized in that: The specific steps of step 2 are as follows: Step 21: Normalize the convolutional data using the following formula: ； ； in, This represents the arithmetic mean of the data set. This indicates the number of samples, that is, the number of observations contained in the dataset; Indicates the first The specific values ​​that each sample can take; This represents the variance of the data set; Indicates the first The deviation of each sample value from the mean; Step 22: Introduce linear offset parameters for transformation reconstruction and These operations perform scale and shift on the simply normalized data, respectively, to transform and reconstruct the data, restoring the feature information of the original data. The specific formulas are as follows: ； ； in, It is a scaling factor that performs a scale operation on the normalized data to adjust the magnitude of the data. As the offset factor, perform a shift operation on the scaled data to adjust the center position of the data; The mean of the data; The variance of the data; It is a small positive number, used to prevent the denominator from being zero; The data is after simple normalization; This is the data after transformation and reconstruction.

Images