Classification Method, Device, Equipment and Storage Medium for Distorted Images

Abstract

The present application discloses a classification method, device, equipment and storage medium for distorted images, which improves the accuracy of classifying distorted images. The method includes: inputting a to-be-tested distorted image into a trained distorted image classification model to obtain a distortion type; training the distorted image classification model in the following manner: inputting each distorted feature vector obtained by extracting the features of training samples using a convolutional neural network into an IRVFL classifier with a regularization coefficient introduced for classification to obtain a predicted distortion type; obtaining an error vector according to the labeled distortion type and the predicted distortion type of each training sample; obtaining a total model risk through the error vector, the regularization coefficient and the output weight vector in the IRVFL classifier; if the total model risk does not meet the specified condition, adjusting the model parameters and the output weight vector, and returning to the step of inputting each distorted feature vector into the IRVFL classifier for classification until the total model risk meets the specified condition, and then ending.

Description

Technical Field

[0001] This application relates to the field of image processing, and particularly to a method, apparatus, device, and storage medium for classifying distorted images. Background Art

[0002] In the context of digital twin technology, video fusion technology is widely used to empower the virtual world and lead twin applications into the era of intelligent Internet of Things and coexistence of virtual and real. Digital twin technology realizes real-time monitoring and prediction of physical systems by mapping data from the physical world to the virtual world. As an important part of this process, video fusion technology provides rich visual information for the virtual world by integrating and processing video data from different sources.

[0003] However, in practical applications, video image frames are usually captured by various types of camera devices, including fisheye cameras, ordinary dome cameras, and bullet cameras, etc. The frames captured by these devices have different degrees of distortion. Specifically, the frames captured by fisheye cameras usually have large radial distortion, resulting in severe deformation of the edge part of the image. The frames captured by ordinary dome cameras and bullet cameras have a smaller degree of distortion, but there is still a certain degree of distortion in some cases. Since the degree of distortion of the frames captured by different types of camera devices is different, if the same distortion correction model is used for correction, the correction effect will be poor, and even serious deviations will occur. Therefore, it is necessary to classify distorted images, so as to select different correction models to improve the accuracy and reliability of distortion correction. Therefore, there is an urgent need for a method for classifying distorted images. Summary of the Invention

[0004] Embodiments of this application provide a method, apparatus, device, and storage medium for classifying distorted images, which are used to classify distorted images and improve the accuracy of classifying distorted images.

[0005] In a first aspect, this application provides a method for classifying distorted images, and the method includes:

[0006] Input the distorted image to be detected into a pre-trained distorted image classification model to obtain the distortion type of the distorted image; wherein, the distorted image classification model includes a convolutional neural network and an IRVFL classifier, and the IRVFL classifier is a classifier obtained by introducing a regularization coefficient into the RVFL classifier;

[0007] Wherein, the distorted image classification model is trained in the following manner:

[0008] Input the training samples into the convolutional neural network for feature extraction to obtain respective distortion feature vectors, wherein the training samples include multiple distorted images and the labeled distortion types corresponding to the multiple distorted images respectively;

[0009] Input each of the distortion feature vectors into the IRVFL classifier for classification to obtain the predicted distortion types corresponding to the respective distortion feature vectors;

[0010] Obtain an error vector based on the labeled distortion type and the predicted distortion type;

[0011] Obtain the total model risk through the error vector, the regularization coefficient, and the output weight vector in the IRVFL classifier;

[0012] If the total model risk does not meet the specified condition, after adjusting the model parameters and the output weight vector in the IRVFL classifier, return to the step of inputting each of the distortion feature vectors into the IRVFL classifier for classification until the total model risk meets the specified condition, and then end the training of the distortion image classification model.

[0013] In the embodiment of the present application, first, the IRVFL classifier obtained by introducing a regularization coefficient into the RVFL classifier is used to classify distorted images. Since the improved IRVFL algorithm determines the total model risk by introducing a regularization coefficient and through the error vector, the regularization coefficient, and the output weight in the IRVFL classifier, it improves the global optimization ability of the algorithm, avoids the algorithm falling into a local optimal solution during the classification process, thereby improving the robustness and generalization ability of the model and enhancing the accuracy of image classification.

[0014] In a possible embodiment, the convolutional neural network is obtained by replacing the ReLU activation function in the residual block of the Resnet50 convolutional neural network with the SELU activation function.

[0015] In the embodiment of the present application, the SELU activation function can avoid the common "dead" neuron problem in the ReLU function, that is, when the input is always negative, the neuron output is always zero. SELU also has a non-zero output in the negative interval, thus avoiding the problem of neuron "death". In addition, the SELU activation function has a self-normalizing property, which can automatically adjust the output distribution of neurons during the training process to keep it within a stable range, and the self-normalizing property reduces the problems of gradient vanishing and gradient explosion during the training process, thereby improving the training efficiency and stability of the model; especially when dealing with complex and diverse distorted image data, SELU can better capture the features of distorted images and improve the classification accuracy.

[0016] In one embodiment, the convolutional neural network includes an initial convolutional layer, a max pooling layer, a first residual block group, a second residual block group, a third residual block group, a fourth residual block group, an average pooling layer, and a fully connected layer;

[0017] Input the training samples into the convolutional neural network for feature extraction to obtain various distortion feature vectors, including:

[0018] Perform a convolution operation on the first training distorted image using the initial convolutional layer to obtain a first feature vector, where the first training distorted image is any one of the distorted images in the training samples;

[0019] Perform max pooling on the first feature vector using the max pooling layer to obtain a second feature vector;

[0020] Perform continuous feature extraction on the second feature vector using multiple residual blocks to obtain a third feature vector, where the multiple residual blocks include the first residual block group, the second residual block group, the third residual block group, and the fourth residual block group;

[0021] Perform average pooling on the third feature vector using the average pooling layer to obtain a fourth feature vector;

[0022] Perform a fully connected operation on the fourth feature vector using the fully connected layer to obtain a distortion feature vector corresponding to the first training distorted image.

[0023] In one embodiment, the training samples are obtained in the following manner:

[0024] Obtain multiple distorted images of different distortion types;

[0025] Perform color correction on each distorted image to obtain each corrected distorted image;

[0026] Perform distortion type balancing on each of the corrected distorted images to obtain each filtered distorted image;

[0027] Divide each of the filtered distorted images into test samples, the training samples, and validation samples.

[0028] In the embodiments of the present application, by first performing color correction on each distorted image, and then performing distortion type balancing on each distorted image before determining the training samples, the accuracy of the obtained training samples is ensured to further improve the classification effect of the model.

[0029] In one embodiment, the performing color correction on each distorted image to obtain each corrected distorted image includes:

[0030] Use color correction technology to preliminarily adjust the colors of the distorted images, and obtain intermediate distorted images;

[0031] Use color constancy algorithms to perform color correction on the intermediate distorted images respectively, and obtain the corrected distorted images.

[0032] In the embodiments of the present application, the distorted images are adjusted respectively through color correction technology combined with color constancy algorithms, which ensures more comprehensive optimization of the quality of the distorted images, improves the brightness, contrast, and color consistency of the distorted images under different lighting conditions, and reduces image distortion. Thereby improving the accuracy and reliability of subsequent processing.

[0033] In one embodiment, before inputting the distorted feature vectors into the IRVFL classifier for classification to obtain the predicted distorted types corresponding to the distorted feature vectors, the method further includes:

[0034] Use the PCA algorithm to reduce the dimensions of the distorted feature vectors, obtain the reduced distorted feature vectors, and determine the reduced distorted feature vectors as the distorted feature vectors.

[0035] In the embodiments of the present application, by reducing the dimensions of the distorted feature vectors, the dimensions of the large data are reduced, thereby reducing the computational complexity and improving the model training efficiency.

[0036] In one embodiment, obtaining the total model risk through the error vector, the regularization coefficient, and the output weight vector in the IRVFL classifier includes:

[0037] Obtain the total model risk through the following formula:

[0038] ;

[0039] where E is the total model risk, C is a preset constant value, is the error vector, is the regularization coefficient, is the output weight vector.

[0040] In one embodiment, after ending the training of the distorted image classification model, the method further includes:

[0041] Input the validation samples into the convolutional neural network for feature extraction to obtain the distorted feature vectors of the distorted images in the validation samples; wherein, the validation samples include multiple distorted images and the labeled distorted types corresponding to the multiple distorted images;

[0042] Input the distortion feature vectors of the distorted images into the trained distorted image classification model to obtain the predicted distortion types of the distorted images in the validation sample;

[0043] According to the labeled distortion types of the distorted images in the validation sample and the predicted distortion types of the distorted images, obtain the accuracy rate of the trained distorted image classification model;

[0044] If the accuracy rate is not greater than the specified threshold, retrain the distorted image classification model.

[0045] In the embodiments of the present application, the accuracy rate of the trained distorted image classification model is verified through a validation sample. If it is determined that the accuracy rate is not greater than the specified threshold, the distorted image classification model needs to be retrained. Thereby, the accuracy of distorted image classification is improved.

[0046] In a second aspect, the present application provides a classification device for distorted images. The device includes:

[0047] A distortion type determination module, configured to input a to-be-detected distorted image into a pre-trained distorted image classification model to obtain the distortion type; the distorted image classification model includes a convolutional neural network and an IRVFL classifier, and the IRVFL classifier is a classifier obtained by introducing a regularization coefficient into the RVFL classifier;

[0048] A training module, configured to train the distorted image classification model in the following manner:

[0049] Input the training samples into the convolutional neural network for feature extraction to obtain respective distortion feature vectors, where the training samples include multiple distorted images and the labeled distortion types corresponding to the multiple distorted images respectively;

[0050] Input the respective distortion feature vectors into the IRVFL classifier for classification to obtain the predicted distortion types corresponding to the respective distortion feature vectors;

[0051] According to the labeled distortion type and the predicted distortion type, obtain an error vector;

[0052] Through the error vector, the regularization coefficient, and the output weight vector in the IRVFL classifier, obtain the total model risk;

[0053] If the total model risk does not meet the specified condition, adjust the model parameters and the output weight vector in the IRVFL classifier, and then return to the step of inputting the respective distortion feature vectors into the IRVFL classifier for classification until the total model risk meets the specified condition, and then end the training of the distorted image classification model.

[0054] In a possible embodiment, the convolutional neural network is obtained by replacing the ReLU activation function in the residual block of the Resnet50 convolutional neural network with the SELU activation function.

[0055] In a possible embodiment, the convolutional neural network includes an initial convolutional layer, a max pooling layer, a first residual block group, a second residual block group, a third residual block group, a fourth residual block group, an average pooling layer, and a fully connected layer;

[0056] The training module is further configured to:

[0057] Perform a convolution operation on the first training distorted image by using the initial convolutional layer to obtain a first feature vector, where the first training distorted image is any one of the distorted images in the training samples;

[0058] Perform max pooling on the first feature vector by using the max pooling layer to obtain a second feature vector;

[0059] Perform continuous feature extraction on the second feature vector by using a plurality of residual blocks to obtain a third feature vector, where the plurality of residual blocks include the first residual block group, the second residual block group, the third residual block group, and the fourth residual block group;

[0060] Perform an average pooling operation on the third feature vector by using the average pooling layer to obtain a fourth feature vector;

[0061] Perform a fully connected operation on the fourth feature vector by using the fully connected layer to obtain a distorted feature vector corresponding to the first training distorted image.

[0062] In a possible embodiment, the device further includes:

[0063] A training sample determination module, configured to obtain the training samples in the following manner:

[0064] Obtain multiple distorted images of different distortion types;

[0065] Perform color correction on each of the distorted images to obtain each corrected distorted image;

[0066] Perform distortion type balancing on each of the corrected distorted images to obtain each filtered distorted image;

[0067] Obtain the training samples according to each of the filtered distorted images.

[0068] In a possible embodiment, the training sample determination module is further configured to:

[0069] Use color correction technology to preliminarily adjust the colors of the distorted images to obtain intermediate distorted images;

[0070] Use color constancy algorithms to perform color correction on the intermediate distorted images respectively to obtain the corrected distorted images.

[0071] In a possible implementation manner, the device further includes:

[0072] A dimensionality reduction module, configured to, before inputting the distorted feature vectors into the IRVFL classifier for classification to obtain the predicted distorted types corresponding to the distorted feature vectors, use the PCA algorithm to perform dimensionality reduction on the distorted feature vectors to obtain the dimensionality-reduced distorted feature vectors, and determine the dimensionality-reduced distorted feature vectors as the distorted feature vectors.

[0073] In a possible implementation manner, the training module is further configured to:

[0074] Obtain the total model risk through the following formula:

[0075] ;

[0076] where E is the total model risk, C is a preset constant value, is the error vector, is the regularization coefficient, is the output weight vector.

[0077] In a possible embodiment, the device further includes:

[0078] A feature extraction module, configured to, after finishing the training of the distorted image classification model, input the validation samples into the convolutional neural network for feature extraction to obtain the distorted feature vectors of the distorted images in the validation samples; wherein, the validation samples include multiple distorted images and the labeled distorted types corresponding to the multiple distorted images;

[0079] An input module, configured to input the distorted feature vectors of the distorted images into the trained distorted image classification model to obtain the predicted distorted types of the distorted images in the validation samples;

[0080] A verification module, configured to obtain the accuracy rate of the trained distorted image classification model according to the labeled distorted types of the distorted images in the validation samples and the predicted distorted types of the distorted images;

[0081] A judgment module, configured to, if the accuracy rate is not greater than a specified threshold, retrain the distorted image classification model.

[0082] In a third aspect, the present application provides an electronic device, including:

[0083] a memory for storing program instructions;

[0084] a processor for calling the program instructions stored in the memory and executing the steps included in the method according to any one of the first aspects according to the obtained program instructions.

[0085] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, the computer program including program instructions which, when executed by a computer, cause the computer to execute the method according to any one of the first aspects.

[0086] In a fifth aspect, the present application provides a computer program product, the computer program product including: computer program code which, when run on a computer, causes the computer to execute the method according to any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 FIG. is one of the flow diagrams of a method for classifying distorted images provided by an embodiment of the present application;

[0088] Figure 2 FIG. is the flow diagram of determining training samples provided by an embodiment of the present application;

[0089] Figure 3 FIG. is the structural diagram of a convolutional neural network provided by an embodiment of the present application;

[0090] Figure 4 FIG. is the flow diagram of feature extraction for a convolutional neural network provided by an embodiment of the present application;

[0091] Figure 5 FIG. is another flow diagram of a method for classifying distorted images provided by an embodiment of the present application;

[0092] Figure 6 FIG. is the schematic diagram of a device for classifying distorted images provided by an embodiment of the present application;

[0093] Figure 7 FIG. is the structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0094] To make the objectives, technical solutions, and advantages of this application more clearly understood, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application. Without conflict, the embodiments in this application and the features in the embodiments can be arbitrarily combined with each other. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0095] In the description and claims of this application and the above accompanying drawings, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order. In addition, the term "including" and any of its variations are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices. "Multiple" in this application can mean at least two, for example, it can be two, three, or more, and there is no limitation in the embodiments of this application.

[0096] In the technical solutions of this application, the acquisition, dissemination, use, etc. of data all comply with the requirements of relevant national laws and regulations.

[0097] Before introducing the classification method for distorted images provided in the embodiments of this application, for the convenience of understanding, first, the technical background of the embodiments of this application will be introduced in detail below.

[0098] Currently, video image frames are usually captured by various types of imaging devices, including fisheye cameras, ordinary dome cameras, and bullet cameras, etc. The frames captured by these devices have different degrees of distortion. Specifically, the frames captured by fisheye cameras usually have significant radial distortion, resulting in severe deformation of the edge parts of the images. The distortion degree of the frames captured by ordinary dome cameras and bullet cameras is relatively small, but there is still a certain degree of distortion in some cases. Since the distortion degrees of the frames captured by different types of imaging devices are different, if the same distortion correction model is used for correction, the correction effect will be poor, and even serious deviations may occur. Therefore, it is necessary to classify distorted images, so as to select different correction models to improve the accuracy and reliability of distortion correction. Therefore, there is an urgent need for a classification method for distorted images.

[0099] Therefore, the present application provides a method for classifying distorted images. The IRVFL classifier obtained by introducing a regularization coefficient into the RVFL classifier is used to classify the distorted images. Since the improved IRVFL algorithm determines the total model risk by introducing a regularization coefficient and through the error vector, the regularization coefficient, and the output weights in the IRVFL classifier, the global optimization ability of the algorithm is improved, and the algorithm is prevented from falling into a local optimal solution during the classification process, thereby improving the robustness and generalization ability of the model and enhancing the accuracy of image classification. Next, the solution of the present disclosure will be introduced in detail with reference to the accompanying drawings.

[0100] As Figure 1 shown, it is a schematic flowchart of the method for classifying distorted images, which specifically includes the following steps:

[0101] Step 101: Input the training samples into the convolutional neural network for feature extraction to obtain respective distorted feature vectors, where the training samples include multiple distorted images and the respective labeled distorted types corresponding to the multiple distorted images;

[0102] First, the method for determining the training samples in the embodiments of the present application will be described. As Figure 2 shown, it is a schematic flowchart of determining the training samples, which specifically may include the following steps:

[0103] Step 201: Obtain multiple distorted images of different distorted types;

[0104] Step 202: Correct the colors of the respective distorted images to obtain the corrected respective distorted images;

[0105] In a possible implementation manner, step 202 may be specifically implemented as: preliminarily adjusting the colors of the respective distorted images by using color correction technology to obtain respective intermediate distorted images; and performing color correction on the respective intermediate distorted images by using a color constancy algorithm to obtain the corrected respective distorted images.

[0106] The color correction technology in the embodiments of the present application may be a gamma correction algorithm, etc. The specific algorithm may be set according to the specific actual situation, and the embodiments of the present application do not limit the color correction technology here.

[0107] Step 203: Balance the distorted types of the corrected respective distorted images to obtain the filtered respective distorted images;

[0108] In the embodiments of the present application, oversampling technology and / or undersampling technology are used to balance the distorted types of the corrected respective distorted images. And in the embodiments of the present application, the pixels of the filtered respective distorted images are adjusted, and the pixels of any one distorted image are adjusted to pixels.

[0109] Step 204: Obtain the training samples according to the filtered distorted images.

[0110] In the embodiment of the present application, 20% of the filtered distorted images are used as test samples. The remaining 80% of the distorted images are sequentially divided into training samples and validation samples at a ratio of 8:2. However, the embodiment of the present application does not limit the way of dividing each sample, and can be specifically set according to the actual situation.

[0111] The convolutional neural network in the embodiment of the present application is obtained by replacing the ReLU activation function in the residual block of the Resnet50 convolutional neural network with the SELU activation function. Among them, formula (1) is the mathematical model of the SELU activation function:

[0112] …… (1);

[0113] Among them, x is the input value of the SELU activation function, is the output value of the SELU activation function, a is a preset first weight value, and b is a preset second weight value.

[0114] In the embodiment of the present application, b is set to 1.0507 and a is set to 1.67326. However, the specific values of b and a in the embodiment of the present application are not limited, and the specific values of b and a can be set according to the specific actual situation.

[0115] As Figure 3 shown, it is a schematic structural diagram of a convolutional neural network. As can be seen from Figure 3 it, the convolutional neural network 300 includes an initial convolutional layer 301, a max pooling layer 302, a first residual block group 303, a second residual block group 304, a third residual block group 305, a fourth residual block group 306, an average pooling layer 307, and a fully connected layer 308. The following will describe step 101 in the embodiment of the present application in detail in combination with the structure of the convolutional neural network in the embodiment of the present application. As Figure 4 described, it is a schematic flow diagram of feature extraction by a convolutional neural network, which specifically may include the following steps:

[0116] Step 401: Perform a convolution operation on the first training distorted image by using the initial convolutional layer to obtain a first feature vector, where the first training distorted image is any one of the distorted images in the training samples;

[0117] The initial convolutional layer in the embodiment of the present application includes 64 convolution kernels. And the size of the obtained first feature vector is .

[0118] Step 402: Perform max pooling on the first feature vector using the max pooling layer to obtain a second feature vector;

[0119] The size of the max pooling layer in the embodiment of the present application is , so the size of the obtained second feature vector is .

[0120] Step 403: Continuously extract features from the second feature vector using multiple residual blocks to obtain a third feature vector, where the multiple residual blocks include the first residual block group, the second residual block group, the third residual block group, and the fourth residual block group;

[0121] The first residual block group in the embodiment of the present application includes four convolutional residual blocks. The four convolutional residual blocks all include a convolutional layer + a batch normalization layer (BN) + a scaled exponential linear unit (SELU). Each convolutional layer contains 64 convolution kernels, 64 convolution kernels, and 256 convolution kernels. The size of the feature vector obtained after passing through the first residual block group is .

[0122] The second residual block group in the embodiment of the present application includes five convolutional residual blocks. The five convolutional residual blocks all include a convolutional layer + a batch normalization layer (BN) + a scaled exponential linear unit (SELU). Each convolutional layer contains 128 convolution kernels, 128 convolution kernels, and 512 convolution kernels. The size of the feature vector obtained after passing through the second residual block group becomes .

[0123] The third residual block group in the embodiment of the present application includes seven residual blocks. Each residual block includes a convolutional layer + a batch normalization layer (BN) + a scaled exponential linear unit (SELU). The convolutional layer of each residual block contains 256 convolution kernels, 256 convolution kernels, and 1024 convolution kernels. The size of the feature vector output by the third residual block group becomes .

[0124] The fourth residual block group in the embodiment of the present application includes four residual blocks. Each residual block includes a convolutional layer + a batch normalization layer (BN) + a scaled exponential linear unit (SELU). The convolutional layer in each residual block contains 512 convolution kernels, 512 convolution kernels, and 2048 Convolution kernel. The size of the feature vector output by the fourth residual block group becomes .

[0125] Step 404: Perform average pooling operation on the third feature vector by using the average pooling layer to obtain a fourth feature vector;

[0126] The size of the average pooling layer in the embodiment of the present application is .

[0127] Step 405: Perform a full connection operation on the fourth feature vector by using the full connection layer to obtain a distortion feature vector corresponding to the first training distorted image.

[0128] The distortion feature vector obtained in the embodiment of the present application has a feature attribute of 2048 dimensions.

[0129] It should be noted that: the structures of the layers in the Resnet50 convolutional neural network in the embodiment of the present application are only for illustrative purposes and do not limit the structures of the layers in the embodiment of the present application.

[0130] In order to improve the training efficiency of the model, in a possible implementation manner, before performing step 102, use the PCA algorithm to reduce the dimension of each of the distortion feature vectors to obtain the reduced-dimensional distortion feature vectors, and determine the reduced-dimensional distortion feature vectors as the distortion feature vectors.

[0131] In the embodiment of the present application, the parameters of the PCA algorithm are set to 0.99, and at least 99% of the principal component variances are retained. The PCA algorithm will automatically calculate the principal component variances according to the input data and select the fewest principal components such that the sum of the variances of these principal components is greater than or equal to the set variance ratio. The low-dimensional features after dimensionality reduction are used as the number of input layer nodes of the improved RVFL, thereby making the data more compact and efficient, optimizing the number of input layer nodes of the improved RVFL, and improving the training efficiency and accuracy of the model.

[0132] Step 102: Input the distortion feature vectors into the IRVFL classifier for classification to obtain the predicted distortion types corresponding to the distortion feature vectors; the IRVFL classifier is a classifier obtained by introducing a regularization coefficient into the RVFL classifier;

[0133] The regularization coefficient in the embodiment of the present application is used to train the IRVFL classifier, so that the model of the trained IRVFL classifier is more accurate.

[0134] Step 103: Obtain an error vector according to the labeled distortion type and the predicted distortion type;

[0135] In one embodiment, step 103 may be specifically implemented as follows: for any one of the distortion feature vectors, subtract the predicted distortion type corresponding to the distortion feature vector from the marked distortion type to obtain an error vector of the distortion feature vector; determine the average value of the error vectors of the respective distortion feature vectors as the error vector.

[0136] In the embodiments of the present application, each distortion type can be represented by different numerical values. Specifically, it can be set according to the actual situation, and the embodiments of the present application do not limit this here. And the method for determining the error vector in the embodiments of the present application is only for illustrative purposes and does not limit the method for determining the error vector in the embodiments of the present application. The specific method for determining the error vector in the embodiments of the present application can be set according to the specific actual situation.

[0137] Step 104: Obtain the total model risk through the error vector, the regularization coefficient, and the output weight vector in the IRVFL classifier; wherein, the total model risk can be obtained through formula (2):

[0138] ……(2);

[0139] where E is the total model risk, C is a preset constant value, is the error vector, is the regularization coefficient, is the output weight vector.

[0140] Step 105: Determine whether the total model risk meets the specified condition. If not, execute step 106; if so, execute step 107;

[0141] The specified condition in the embodiments of the present application is to minimize the value of the total model risk.

[0142] Step 106: After adjusting the model parameters and the output weight vector in the IRVFL classifier, return to execute step 102;

[0143] In the embodiments of the present application, it is necessary to adjust the output weight vector to the optimal. Among them, the optimal output weight vector can be obtained through formula (3):

[0144] ……(3);

[0145] where, is the optimal output weight vector, H is a preset target output matrix, Y is the actual output matrix in the IRVFL classifier, and C is a preset constant value.

[0146] Step 107: End the training of the distortion image classification model;

[0147] In the embodiment of the present application, after the training of the classification model for distorted images is completed, a validation sample is used to validate the trained classification model for distorted images. If the validation fails, the classification model for distorted images needs to be retrained. If the validation passes, a test sample is used to test the trained classification model for distorted images to determine the performance of the classification model for distorted images. Next, the method for validating the trained classification model for distorted images in the embodiment of the present application will be introduced. In a possible embodiment, the validation sample is input into the convolutional neural network for feature extraction to obtain the distortion feature vectors of each distorted image in the validation sample; wherein, the validation sample includes multiple distorted images and the corresponding labeled distortion types; the distortion feature vectors of each distorted image are input into the trained classification model for distorted images to obtain the predicted distortion types of each distorted image in the validation sample; according to the labeled distortion types of each distorted image in the validation sample and the predicted distortion types of each distorted image, the accuracy rate of the trained classification model for distorted images is obtained; if the accuracy rate is not greater than the specified threshold, the classification model for distorted images is retrained; if the accuracy rate is greater than the specified threshold, the training of the classification model for distorted images is ended.

[0148] It should be noted that: the specified threshold in the embodiment of the present application can be set according to specific actual situations, and the embodiment of the present application does not limit the specific value of the specified threshold here.

[0149] Step 108: Input the distorted image to be detected into the pre-trained classification model for distorted images to obtain the distortion type of the distorted image; wherein, the classification model for distorted images includes a convolutional neural network and an IRVFL classifier.

[0150] The distortion types in the embodiment of the present application include radial distortion, barrel distortion, pincushion distortion, and mixed distortion, etc.

[0151] To further understand the classification method for distorted images in the present application, as Figure 5 shown, it is a schematic flow diagram of the classification method for distorted images, which specifically includes the following steps:

[0152] Step 501: Obtain multiple distorted images with different distortion types;

[0153] Step 502: Correct the colors of each distorted image to obtain the corrected distorted images;

[0154] Step 503: Balance the distortion types of the corrected distorted images to obtain the filtered distorted images;

[0155] Step 504: Obtain the training samples according to the filtered distorted images;

[0156] Step 505: Input the training samples into the convolutional neural network for feature extraction to obtain respective distorted feature vectors. Among them, the training samples include multiple distorted images and the marked distorted types corresponding to the multiple distorted images respectively. The convolutional neural network is obtained by replacing the ReLU activation function in the residual block of the Resnet50 convolutional neural network with the SELU activation function;

[0157] Step 506: Use the PCA algorithm to reduce the dimension of the respective distorted feature vectors to obtain the dimension-reduced respective distorted feature vectors, and determine the dimension-reduced respective distorted feature vectors as the respective distorted feature vectors;

[0158] Step 507: Input the respective distorted feature vectors into the IRVFL classifier for classification to obtain the predicted distorted types corresponding to the respective distorted feature vectors;

[0159] Step 508: Obtain an error vector according to the labeled distorted type and the predicted distorted type;

[0160] Step 509: Obtain the total model risk through the error vector, the regularization coefficient, and the output weight vector in the IRVFL classifier;

[0161] Step 510: Determine whether the total model risk meets the specified condition. If so, execute Step 512; if not, execute Step 511;

[0162] Step 511: After adjusting the model parameters and the output weight vector in the IRVFL classifier, return to execute Step 507;

[0163] Step 512: End the training of the distorted image classification model;

[0164] Step 513: Input the distorted image to be detected into the pre-trained distorted image classification model to obtain the distorted type of the distorted image.

[0165] Based on the same inventive concept, an embodiment of the present application provides a classification device for distorted images. The effect of the classification device for distorted images is similar to that of the foregoing method, and will not be elaborated here.

[0166] Figure 6 It is a schematic structural diagram of a classification device for distorted images according to an embodiment of the present disclosure.

[0167] Such as Figure 6As shown in the figure, the distortion image classification device 600 of the present disclosure may include a distortion type determination module 610 and a training module 620.

[0168] The distortion type determination module 610 is configured to input the distortion image to be detected into a pre-trained distortion image classification model to obtain the distortion type; the distortion image classification model includes a convolutional neural network and an IRVFL classifier, and the IRVFL classifier is a classifier obtained by introducing a regularization coefficient into the RVFL classifier;

[0169] The training module 620 is configured to train the distortion image classification model in the following manner:

[0170] Input the training samples into the convolutional neural network for feature extraction to obtain respective distortion feature vectors, where the training samples include multiple distortion images and the respective labeled distortion types corresponding to the multiple distortion images;

[0171] Input the respective distortion feature vectors into the IRVFL classifier for classification to obtain the predicted distortion types corresponding to the respective distortion feature vectors;

[0172] According to the labeled distortion type and the predicted distortion type, obtain an error vector;

[0173] Through the error vector, the regularization coefficient, and the output weight vector in the IRVFL classifier, obtain the total model risk;

[0174] If the total model risk does not meet the specified condition, then after adjusting the model parameters and the output weight vector in the IRVFL classifier, return to the step of inputting the respective distortion feature vectors into the IRVFL classifier for classification until the total model risk meets the specified condition, and then end the training of the distortion image classification model.

[0175] In a possible embodiment, the convolutional neural network is obtained by replacing the ReLU activation function in the residual block of the Resnet50 convolutional neural network with the SELU activation function.

[0176] In a possible embodiment, the convolutional neural network includes an initial convolutional layer, a max pooling layer, a first residual block group, a second residual block group, a third residual block group, a fourth residual block group, an average pooling layer, and a fully connected layer;

[0177] The training module 620 is further configured to:

[0178] Perform a convolution operation on the first training distorted image using the initial convolution layer to obtain a first feature vector, where the first training distorted image is any one of the distorted images in the training samples;

[0179] Perform max pooling on the first feature vector using the max pooling layer to obtain a second feature vector;

[0180] Perform continuous feature extraction on the second feature vector using multiple residual blocks to obtain a third feature vector, where the multiple residual blocks include the first residual block group, the second residual block group, the third residual block group, and the fourth residual block group;

[0181] Perform average pooling on the third feature vector using the average pooling layer to obtain a fourth feature vector;

[0182] Perform a fully connected operation on the fourth feature vector using the fully connected layer to obtain a distorted feature vector corresponding to the first training distorted image.

[0183] In a possible embodiment, the apparatus further includes:

[0184] A training sample determination module 630, configured to obtain the training samples in the following manner:

[0185] Obtain multiple distorted images of different distortion types;

[0186] Perform color correction on each distorted image to obtain each corrected distorted image;

[0187] Perform distortion type balancing on each of the corrected distorted images to obtain each filtered distorted image;

[0188] Obtain the training samples according to each of the filtered distorted images.

[0189] In a possible embodiment, the training sample determination module 630 is further configured to:

[0190] Perform a preliminary adjustment on the colors of the distorted images using color correction technology to obtain each intermediate distorted image;

[0191] Perform color correction on each of the intermediate distorted images using a color constancy algorithm to obtain each of the corrected distorted images.

[0192] In a possible implementation manner, the apparatus further includes:

[0193] A dimensionality reduction module 640, which is used to perform dimensionality reduction on each of the distorted feature vectors by using the PCA algorithm before inputting the distorted feature vectors into the IRVFL classifier for classification to obtain the predicted distorted types corresponding to the distorted feature vectors, and determining the dimensionality-reduced distorted feature vectors as the distorted feature vectors.

[0194] In a possible implementation manner, the training module 620 is further configured to:

[0195] Obtain the total model risk through the following formula:

[0196] ;

[0197] where E is the total model risk, C is a preset constant value, is the error vector, is the regularization coefficient, is the output weight vector.

[0198] In a possible embodiment, the device further includes:

[0199] A feature extraction module 650, which is used to input a validation sample into the convolutional neural network for feature extraction after the training of the distorted image classification model is completed, to obtain the distorted feature vectors of each distorted image in the validation sample; where the validation sample includes multiple distorted images and the labeled distorted types corresponding to the multiple distorted images;

[0200] An input module 660, which is used to input the distorted feature vectors of each distorted image into the trained distorted image classification model to obtain the predicted distorted types of each distorted image in the validation sample;

[0201] A verification module 670, which is used to obtain the accuracy rate of the trained distorted image classification model according to the labeled distorted types of each distorted image in the validation sample and the predicted distorted types of each distorted image;

[0202] A judgment module 680, which is used to retrain the distorted image classification model if the accuracy rate is not greater than a specified threshold.

[0203] Based on the same technical concept, an embodiment of the present application further provides an electronic device 700, as Figure 7 shown, including at least one processor 701 and a memory 702 connected to at least one processor. In the embodiment of the present application, the specific connection medium between the processor 701 and the memory 702 is not limited. Figure 7Take the connection between the central processor 701 and the memory 702 through the bus 703 as an example. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 7 it is only represented by a thick line in the figure, but it does not mean that there is only one bus or one type of bus.

[0204] Among them, the processor 701 is the control center of the electronic device. It can connect various parts of the electronic device through various interfaces and circuits, and realize data processing by running or executing the instructions stored in the memory 702 and calling the data stored in the memory 702. Optionally, the processor 701 may include one or more processing units. The processor 701 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interfaces, and application programs, etc., and the modem processor mainly processes the issued instructions. It can be understood that the above-mentioned modem processor may not be integrated into the processor 701 either. In some embodiments, the processor 701 and the memory 702 may be implemented on the same chip. In some embodiments, they may also be separately implemented on independent chips.

[0205] The processor 701 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the embodiment of the system problem location method may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0206] The memory 702, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 702 can include at least one type of storage medium. For example, it can include flash memory, hard disks, multimedia cards, card-type memories, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memories, magnetic disks, optical discs, and so on. The memory 702 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 702 in the embodiments of the present application can also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data.

[0207] In the embodiments of the present application, the memory 702 stores a computer program. When the program is executed by the processor 701, the processor 701 is caused to execute the classification method of the distorted image described above.

[0208] Since this electronic device is the electronic device in the method of the embodiments of the present application, and the principle by which this electronic device solves problems is similar to that of the method, the implementation of this electronic device can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0209] Based on the same inventive concept, the embodiments of the present application provide a computer-readable storage medium. The computer program product includes: computer program code. When the computer program code runs on a computer, the computer is caused to execute the classification method of the distorted image as described in any of the above. Therefore, the implementation of the above computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0210] Based on the same inventive concept, the embodiments of the present application further provide a computer program product. The computer program product includes: computer program code. When the computer program code runs on a computer, the computer is caused to execute the classification method of the distorted image as described in any of the above. Since the principle by which the above computer program product solves problems is similar to that of the classification method of the distorted image, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0211] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0212] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0213] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0214] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of user operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0215] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.

Claims

1. A method for classifying distorted images, characterized in that: The method comprises: Inputting the distorted image to be detected into a pre-trained distorted image classification model to obtain the distortion type of the distorted image; wherein the distorted image classification model includes a convolutional neural network and an IRVFL classifier, and the IRVFL classifier is a classifier obtained by introducing a regularization coefficient into the RVFL classifier; The distorted image classification model is trained in the following manner: Inputting the training samples into the convolution neural network for feature extraction to obtain each distortion feature vector, wherein the training samples include a plurality of distorted images and annotated distortion types corresponding to the plurality of distorted images; Inputting each distortion feature vector into the IRVFL classifier for classification to obtain a predicted distortion type corresponding to each distortion feature vector; Obtaining an error vector according to the marked distortion type and the predicted distortion type; Obtaining a total model risk through the error vector, the regularization coefficient, and the output weight vector in the IRVFL classifier; If the total model risk does not meet the specified conditions, the model parameters and output weight vectors in the IRVFL classifier are adjusted, and then the step of inputting each distortion feature vector into the IRVFL classifier for classification is returned until the total model risk meets the specified conditions, and the training of the distorted image classification model is terminated.

2. The method according to claim 1, characterized in that The convolutional neural network is obtained by replacing the ReLU activation function in the residual block in the Resnet50 convolutional neural network with the SELU activation function.

3. The method according to claim 2, characterized in that The convolution neural network includes an initial convolution layer, a maximum pooling layer, a first residual block group, a second residual block group, a third residual block group, a fourth residual block group, an average pooling layer and a fully connected layer; The training samples are input into the convolutional neural network for feature extraction to obtain the distortion feature vectors, including: Using the initial convolution layer to perform a convolution operation on a first training distorted image to obtain a first feature vector, wherein the first training distorted image is any distorted image in the training samples; Performing maximum pooling on the first feature vector using the maximum pooling layer to obtain a second feature vector; Performing continuous feature extraction on the second feature vector using a plurality of residual blocks to obtain a third feature vector, wherein the plurality of residual blocks include the first residual block group, the second residual block group, the third residual block group, and the fourth residual block group; Using the average pooling layer to perform an average pooling operation on the third eigenvector to obtain a fourth eigenvector; The fully connected layer is used to perform a fully connected operation on the fourth feature vector to obtain a distorted feature vector corresponding to the first training distorted image.

4. The method according to claim 1, characterized in that: The training samples are obtained by: Acquire multiple distorted images with different distortion types; Correcting the color of each distorted image to obtain each corrected distorted image; Performing distortion type balancing on the corrected distorted images to obtain filtered distorted images; The training samples are obtained according to the filtered distorted images.

5. The method according to claim 4, characterized in that Correcting the colors of the distorted images to obtain the corrected distorted images includes: Preliminarily adjusting the colors of the distorted images by using a color correction technology to obtain intermediate distorted images; The color of each intermediate distorted image is respectively corrected by using a color constancy algorithm to obtain each corrected distorted image.

6. The method according to claim 1, characterized in that Before inputting each distortion feature vector into the IRVFL classifier for classification to obtain the predicted distortion type corresponding to each distortion feature vector, the method further includes: The PCA algorithm is used to reduce the dimension of each distorted feature vector to obtain each distorted feature vector after the dimension reduction, and each distorted feature vector after the dimension reduction is determined as the distorted feature vector.

7. The method according to claim 1, characterized in that The total model risk is obtained by using the error vector, the regularization coefficient and the output weight vector in the IRVFL classifier, including: The total model risk is obtained by the following formula: ； Wherein, E is the total model risk, C is a preset constant value, is the error vector, is the regularization coefficient, is the output weight vector.

8. The method according to claim 1, characterized in that: After completing the training of the distorted image classification model, the method further includes: Inputting the verification sample into the convolutional neural network for feature extraction to obtain a distortion feature vector of each distorted image in the verification sample; wherein the verification sample includes a plurality of distorted images and annotated distortion types corresponding to the plurality of distorted images; Inputting the distortion feature vector of each distorted image into the trained distorted image classification model to obtain the predicted distortion type of each distorted image in the verification sample; Obtaining the accuracy of the trained distorted image classification model according to the annotated distortion type of each distorted image in the verification sample and the predicted distortion type of each distorted image; If the accuracy is not greater than a specified threshold, the distorted image classification model is retrained.

9. A distorted image classification device, characterized in that: The device comprises: A distortion type determination module, used for inputting a distorted image to be detected into a pre-trained distorted image classification model to obtain the distortion type of the distorted image; the distorted image classification model includes a convolutional neural network and an IRVFL classifier, and the IRVFL classifier is a classifier obtained by introducing a regularization coefficient into an RVFL classifier; The training module is used to train the distorted image classification model in the following manner: Inputting the training samples into the convolution neural network for feature extraction to obtain each distortion feature vector, wherein the training samples include a plurality of distorted images and annotated distortion types corresponding to the plurality of distorted images; Inputting each distortion feature vector into the IRVFL classifier for classification to obtain a predicted distortion type corresponding to each distortion feature vector; Obtaining an error vector according to the marked distortion type and the predicted distortion type; Obtaining a total model risk through the error vector, the regularization coefficient, and the output weight vector in the IRVFL classifier; If the total model risk does not meet the specified conditions, the model parameters and output weight vectors in the IRVFL classifier are adjusted, and then the step of inputting each distortion feature vector into the IRVFL classifier for classification is returned until the total model risk meets the specified conditions, and the training of the distorted image classification model is terminated.

10. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the methods of claims 1-8 according to the obtained program instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 8.

Images