Image processing method, system, device and computer-readable storage medium

Abstract

The present application discloses an image processing method, system, device and computer-readable storage medium, which obtains a target image to be processed; obtains a feature image of a convolutional hidden layer of a trained source model; uses the feature image of the last layer as a depth feature and the remaining feature images as low-level features; extracts a feature vector of interest of the low-level feature for each low-level feature; trains a target model based on the feature vector of interest and the depth feature to obtain a trained target model; processes a target image based on the target model to obtain a processing result. In the present application, the target model can be trained based on the feature vector of interest and the depth feature of the low-level feature. On the basis of retaining feature information at multiple levels, the number of features during target model training is increased, the feature robustness during target model training is enhanced, the training effect of the target model is enhanced, and the processing effect of the target model on the target image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to an image processing method, system, device, and computer-readable storage medium. Background Art

[0002] Image classification and recognition methods based on deep learning are playing a vital role in a growing number of application areas. The powerful feature extraction and nonlinear data fitting capabilities of the CNN architecture enable excellent results for both general classification problems and specific pattern recognition problems, making deep learning a viable approach for image processing. This is due not only to the deep neural network architecture, but also to the large amount of image data required to achieve highly robust models. However, the large amount of data training and the complex model parameter learning also lead to reduced model training efficiency. Improving model learning efficiency has become a pressing issue, especially in increasingly complex application scenarios.

[0003] Transfer learning is a method of applying knowledge learned from one environment to learning tasks in a new environment. Typically, when the source scene has sufficient data and the target scene has less, transfer learning can be used to transfer the source model to the target model, enabling the target model to achieve high-performance and efficient learning. Transfer learning methods use a small number of samples to learn in a new scenario based on a pre-trained neural network model, which can reduce the need to retrain some network parameters and improve model learning efficiency. For example, the convolutional hidden layers of the source model can be kept unchanged, while adding several convolutional layers, pooling layers, salient region preprocessing, and background preprocessing to extract more shallow features, increase the feature vector, and improve overfitting, thereby ensuring the effectiveness of transfer learning.

[0004] However, the existing transfer learning uses relatively few samples, which makes it difficult to ensure the effectiveness of transfer learning, thereby affecting the image processing effect.

[0005] In summary, how to improve image processing effects is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0006] The purpose of this application is to provide an image processing method that can, to a certain extent, solve the technical problem of how to improve the image processing effect. This application also provides an image processing system, device and computer-readable storage medium.

[0007] In order to achieve the above objectives, this application provides the following technical solutions:

[0008] An image processing method, comprising:

[0009] Obtain the target image to be processed;

[0010] Get the feature image of the convolutional hidden layer of the trained source model;

[0011] The feature image of the last layer is used as a deep feature, and the remaining feature images are used as low-level features;

[0012] For each of the low-level features, extracting a feature vector of interest of the low-level feature;

[0013] Training a target model based on the feature vector of interest and the depth feature to obtain the trained target model;

[0014] The target image is processed based on the target model to obtain a processing result.

[0015] Preferably, extracting the interesting feature vector of the low-level feature includes:

[0016] Divide each image in the low-level features into corresponding rectangular sub-regions;

[0017] Calculating a feature descriptor for each of the rectangular sub-regions;

[0018] For each rectangular sub-region, based on the feature descriptor, calculating the correlation value between the rectangular sub-regions of every two images in the low-level features, selecting the two images with the largest correlation values ​​as the feature images of the rectangular sub-region, and taking the average value of the feature descriptors of the rectangular sub-region in the feature images as the feature of interest of the rectangular sub-region;

[0019] The features of interest in all the rectangular sub-regions are used as the feature of interest vector.

[0020] Preferably, calculating the feature descriptor of each rectangular sub-region includes:

[0021] For each rectangular sub-region, a grayscale histogram, a gradient direction histogram, and an LBP histogram of the rectangular sub-region are calculated, and the grayscale histogram, the gradient direction histogram, and the LBP histogram are connected to obtain the feature descriptor.

[0022] Preferably, the training of the target model based on the feature vector of interest and the depth feature includes:

[0023] Fusing all the feature vectors of interest with the depth features to obtain the target features;

[0024] determining a target type of the target feature;

[0025] The target model is trained based on the target features and the target type.

[0026] Preferably, determining the target type of the target feature includes:

[0027] The target features are processed based on a DNN classification model to obtain the target type.

[0028] Preferably, the DNN classification model includes three fully connected layers and one softmax layer.

[0029] Preferably, obtaining the target image to be processed includes:

[0030] Obtain the initial image to be processed;

[0031] The initial image is cropped and / or interpolated and / or brightness normalized to obtain the target image.

[0032] An image processing system, comprising:

[0033] A first acquisition module is used to acquire a target image to be processed;

[0034] The second acquisition module is used to obtain the feature image of the convolution hidden layer of the trained source model;

[0035] A first setting module is used to use the feature image of the last layer as a deep feature and the remaining feature images as low-level features;

[0036] A first extraction module is configured to extract, for each of the low-level features, a feature vector of interest of the low-level feature;

[0037] A first training module is used to train the target model based on the feature vector of interest and the depth feature to obtain the trained target model;

[0038] The first processing module is used to process the target image based on the target model to obtain a processing result.

[0039] An image processing device, comprising:

[0040] memory for storing computer programs;

[0041] A processor is configured to implement the steps of any of the above-mentioned image processing methods when executing the computer program.

[0042] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-mentioned image processing methods.

[0043] The present application provides an image processing method, which comprises the following steps: obtaining a target image to be processed; obtaining a feature image of a convolutional hidden layer of a trained source model; using the feature image of the last layer as a depth feature and the remaining feature images as low-level features; extracting a feature vector of interest of the low-level feature for each low-level feature; training a target model based on the feature vector of interest and the depth feature to obtain a trained target model; and processing the target image based on the target model to obtain a processing result. In the present application, the feature image of the convolutional hidden layer of the source model can be classified into depth features and low-level features, and the feature vector of interest of the low-level feature can be additionally extracted. Finally, the target model is trained based on the feature vector of interest and the depth feature. On the basis of retaining the feature information of multiple levels of the source model, the number of features of the target model during training is increased, the feature robustness of the target model during training is enhanced, and the training effect of the target model can be enhanced. Accordingly, the processing effect of the target model on the target image can be improved. The present application provides an image processing system, device, and computer-readable storage medium, which also solves corresponding technical problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0045] Figure 1 A first flow chart of an image processing method provided in an embodiment of the present application;

[0046] Figure 2 A second flow chart of an image processing method provided in an embodiment of the present application;

[0047] Figure 3 Flowchart for extracting interesting features from the feature image on convolutional layer 7 in the VGG16 model;

[0048] Figure 4 Schematic diagram for generating feature vectors of interest;

[0049] Figure 5 This is the model structure diagram of the DNN classifier;

[0050] Figure 6 A schematic diagram of the structure of an image processing system provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0052] Figure 8 Another structural schematic diagram of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] See also Figure 1 , Figure 1 This is a first flow chart of an image processing method provided in an embodiment of the present application.

[0055] An image processing method provided in an embodiment of the present application may include the following steps:

[0056] Step S101: Acquire a target image to be processed.

[0057] In actual applications, the target image to be processed can be obtained first. The type of the target image can be determined according to actual needs. For example, the target image can be a pedestrian image, a car image, a server image, etc. This application does not make specific limitations here.

[0058] In specific application scenarios, when obtaining a target image to be processed, an initial image to be processed can be obtained first, and then the initial image can be processed, such as by cropping and / or interpolation and / or brightness normalization, to obtain a target image. It should be noted that the cropping operation can increase the randomness of the image data and achieve data enhancement; the interpolation operation requires adjusting the image size to the image size input to the transfer neural network model (source model). Taking the VGG16 model as an example, the image needs to be interpolated to (224,224); the brightness normalization operation normalizes the image based on the image brightness mean and variance, which can make the neural network model more easily converged; that is, the initial image can be processed as needed to obtain a target image that is easier for the target model to converge, thereby improving the training efficiency of the target model and subsequently improving the processing efficiency of the target image.

[0059] Step S102: Obtain the feature image of the convolution hidden layer of the trained source model.

[0060] In practical applications, after obtaining the target image to be processed, the feature image of the convolutional hidden layer of the trained source model can be obtained, so that the target model can be subsequently transferred based on the feature image of the convolutional hidden layer of the source model.

[0061] It should be noted that the source model refers to a trained model for processing the target image, the target model refers to an untrained model for processing the target image, and some features of the source model can be used to train the target model to speed up the training efficiency of the target model. The source model involved in this application can be AlexNet, VGGNet, GoogleNet, ResNet, VGG16 model, etc., and the function of the source model can be determined according to actual needs, and this application does not make specific limitations here.

[0062] Step S103: The feature image of the last layer is used as the deep feature, and the remaining feature images are used as low-level features.

[0063] In practical applications, after obtaining the feature image of the convolutional hidden layer of the trained source model, the feature image of the last convolutional hidden layer of the source model can be used as the deep feature, and the feature images of the remaining convolutional hidden layers can be used as the low-level features.

[0064] It should be noted that the convolutional hidden layer of the source model can be determined according to the type of the source model. For ease of understanding, take the VGG16 model as an example. The VGG16 model has a total of 38 layers, layers 0 to 30 correspond to convolutional hidden layers, and layers 31 to 37 are fully connected hidden layers. The actual network structure of the convolutional hidden layer can be shown as follows:

[0065] Layer 0: Convolutional layer, input image size is (3,224,224), output size is (64,224,224), and convolution kernel size is (3,3);

[0066] Layer 1: Activation layer, ReLU activation function, input and output image sizes are both (64, 224, 224);

[0067] Layer 2: Convolutional layer, input image size is (64, 224, 224), output size is (64, 224, 224), and convolution kernel size is (3, 3);

[0068] Layer 3: Activation layer, ReLU activation function, input and output image sizes are both (64, 224, 224);

[0069] Layer 4: Pooling layer, maxpool pooling, input image size is (64, 224, 224), output size is (64, 112, 112), pooling kernel size is 2;

[0070] 5 layers: convolutional layer, input image size is (64,112,112), output image size is (128,112,112), convolution kernel size is (3,3);

[0071] 6th layer: Activation layer, ReLU activation function, input and output image sizes are both (128, 112, 112);

[0072] 7 layers: convolutional layer, input image size is (128,112,112), output image size is (128,112,112), convolution kernel size is (3,3);

[0073] 8th layer: Activation layer, ReLU activation function, input and output image sizes are both (128, 112, 112);

[0074] Layer 9: Pooling layer, maxpool pooling, input image size is (128, 112, 112);, output size is (128, 56, 56), pooling kernel size is 2;

[0075] Layer 10: Convolutional layer, input image size is (128, 56, 56), output image size is (256, 56, 56), and convolution kernel size is (3, 3);

[0076] Layer 11: Activation layer, ReLU activation function, input and output image sizes are both (256, 56, 56);

[0077] Layer 12: Convolutional layer, input image size is (256, 56, 56), output image size is (256, 56, 56), and convolution kernel size is (3, 3);

[0078] Layer 13: Activation layer, ReLU activation function, input and output image sizes are both (256, 56, 56);

[0079] Layer 14: Convolutional layer, input image size is (256, 56, 56), output image size is (256, 56, 56), and convolution kernel size is (3, 3);

[0080] Layer 15: Activation layer, ReLU activation function, input and output image sizes are both (256, 56, 56);

[0081] Layer 16: Pooling layer, maxpool pooling, input image size is (256, 56, 56), output size is (256, 28, 28), pooling kernel size is 2;

[0082] Layer 17: Convolutional layer, input image size is (256, 28, 28), output image size is (512, 28, 28), and convolution kernel size is (3, 3);

[0083] Layer 18: Activation layer, ReLU activation function, input and output image sizes are both (512, 28, 28);

[0084] Layer 19: Convolutional layer, input image size is (512, 28, 28), output image size is (512, 28, 28), and convolution kernel size is (3, 3);

[0085] Layer 20: Activation layer, ReLU activation function, input and output image sizes are both (512, 28, 28);

[0086] Layer 21: Convolutional layer, input image size is (512, 28, 28), output image size is (512, 28, 28), and convolution kernel size is (3, 3);

[0087] Layer 22: Activation layer, ReLU activation function, input and output image sizes are both (512, 28, 28);

[0088] Layer 23: Pooling layer, maxpool pooling, input image size is (512, 28, 28), output size is (512, 14, 14), pooling kernel size is 2;

[0089] 24 layers: convolutional layer, input image size is (512,14,14), output image size is (512,14,14), convolution kernel size is (3,3);

[0090] Layer 25: Activation layer, ReLU activation function, input and output image sizes are both (512, 14, 14);

[0091] 26 layers: convolutional layer, input image size is (512,14,14), output image size is (512,14,14), convolution kernel size is (3,3);

[0092] Layer 27: Activation layer, ReLU activation function, input and output image sizes are both (512, 14, 14);

[0093] 28 layers: convolutional layer, input image size is (512,14,14), output image size is (512,14,14), convolution kernel size is (3,3);

[0094] Layer 29: Activation layer, ReLU activation function, input and output image sizes are both (512, 14, 14);

[0095] Layer 30: Pooling layer, maxpool pooling, input image size is (512, 14, 14), output size is (512, 7, 7), pooling kernel size is 2;

[0096] Correspondingly, the deep features of the VGG16 model are the feature images output by the 30th pooling layer; the low-level features of the VGG16 model are the feature images output by other convolutional layers, and the convolutional layers corresponding to other layers are: 0, 2, 5, 7, 10, 12, 14, 17, 19, 21, 24, 26, and 28.

[0097] Step S104: For each low-level feature, extract the interesting feature vector of the low-level feature.

[0098] In practical applications, after taking the feature image of the last layer as the deep feature and the remaining feature images as the low-level features, the feature vector of interest of the low-level feature can be extracted for each low-level feature, that is, the feature vector of interest of each low-level feature can be extracted. In this way, as many feature vectors of interest as there are low-level features, because the feature vector of interest is a newly added parameter for training the target model, the present application can increase the training parameters of the target model without increasing the sample size, thereby ensuring the training effect of the subsequent training model.

[0099] Step S105: Train the target model based on the feature vector of interest and the depth feature to obtain a trained target model.

[0100] Step S106: Process the target image based on the target model to obtain a processing result.

[0101] In practical applications, after extracting the feature vector of interest for each low-level feature, the target model can be trained based on the feature vector of interest and the deep features to obtain a trained target model, and the target image can be processed based on the target model to obtain the processing result.

[0102] It should be noted that in the process of training the target model based on the feature vector of interest and the depth feature, training samples must also be applied. The specific training sample type, quantity, training method, etc. can be flexibly adjusted according to actual needs, and this application does not make specific limitations here.

[0103] The present application provides an image processing method, which obtains a target image to be processed; obtains a feature image of a convolution hidden layer of a trained source model; uses the feature image of the last layer as a depth feature and the remaining feature images as low-level features; for each low-level feature, extracts a feature vector of interest of the low-level feature; trains a target model based on the feature vector of interest and the depth feature to obtain a trained target model; processes the target image based on the target model to obtain a processing result. In the present application, the feature image of the convolution hidden layer of the source model can be classified into depth features and low-level features, and the feature vector of interest of the low-level feature can be additionally extracted. Finally, the target model is trained based on the feature vector of interest and the depth feature. On the basis of retaining the feature information of multiple levels of the source model, the number of features during target model training is increased, the feature robustness of the target model during training is enhanced, and then the training effect of the target model can be enhanced. Accordingly, the processing effect of the target model on the target image can be improved.

[0104] See also Figure 2 , Figure 2 This is a second flow chart of an image processing method provided in an embodiment of the present application.

[0105] An image processing method provided in an embodiment of the present application may include the following steps:

[0106] Step S201: Acquire a target image to be processed.

[0107] Step S202: Obtain the feature image of the convolution hidden layer of the trained source model.

[0108] Step S203: The feature image of the last layer is used as the deep feature, and the remaining feature images are used as low-level features.

[0109] Step S204: For each low-level feature, each image in the low-level feature is divided into corresponding rectangular sub-regions; and a feature descriptor of each rectangular sub-region is calculated.

[0110] In practical applications, in the process of extracting the feature vector of interest of low-level features, each image in the low-level features can be divided into corresponding rectangular sub-regions, and the feature descriptor of each rectangular sub-region can be calculated, so that the feature vector of interest of the low-level features can be determined based on the feature descriptor of the rectangular sub-region.

[0111] In specific application scenarios, when dividing each image in the low-level features into corresponding rectangular sub-regions, the number of rectangular sub-regions can be flexibly determined according to actual needs. For example, in the VGG16 model, the convolution layer 7 of the VGG16 model outputs 128 112×112 feature images. For each image, the size of the rectangular sub-region should be consistent with the output image (maxpool-30) of the last pooling layer of the VGG16 model. According to the network parameters of the VGG16 model, the size of the rectangular sub-region is 7×7, so for a 112×112 feature image, it can be divided into (16×16) rectangular sub-regions.

[0112] In a specific application scenario, in the process of calculating the feature descriptor of each rectangular sub-region, for each rectangular sub-region, the grayscale histogram, gradient direction histogram, and LBP (Local Binary Patterns) histogram of the rectangular sub-region can be calculated, and the grayscale histogram, gradient direction histogram, and LBP histogram are connected to obtain a feature descriptor. In this process, the grayscale histogram can describe the grayscale distribution in the local area, the gradient direction histogram can describe the grayscale change amplitude and direction distribution in the local area, and the LBP histogram can describe the grayscale change direction in the local area. The feature descriptor generated in this way can describe the grayscale information of the matrix sub-region in detail, making it easier for the subsequent generated feature vector of interest to describe the grayscale information in the low-level features in detail.

[0113] In a specific application scenario, when calculating the grayscale histogram of a rectangular sub-region, the grayscale of the image in the matrix sub-region can be quantized first. The quantization number is recorded as K. The image grayscale value is normalized to [0, 1] and then quantized to [0, K-1]. For the pixel (x, y) on the feature image I, the grayscale value is recorded as I(x, y). The grayscale quantization formula is as follows:

[0114]

[0115] Where min(I) and max(I) represent the minimum and maximum values ​​of the feature image I, respectively;

[0116] Calculate the grayscale distribution of each rectangular sub-region on the feature image I, that is, count the proportion of the number of pixels of each gray level in the rectangular sub-region to the total number of pixels in the sub-region. For the i-th rectangular region, the grayscale histogram feature is recorded as The calculation formula is as follows:

[0117]

[0118] in, It represents the grayscale value of the pixel in the i-th rectangular area after quantization, and its value range is {0, 1, ..., K-1}. rSize represents the size of the rectangular sub-area, that is, the number of pixels in the rectangular sub-area. Represents the grayscale distribution probability within the rectangular sub-region and describes the grayscale distribution feature information.

[0119] In a specific application scenario, in the process of calculating the gradient direction histogram of a rectangular sub-region, Sobel edge detection can be performed on the image first, and the (3×3) convolution kernels kx and ky are used to perform convolution operations on the feature image. dx represents the calculation of the gradient sobelX in the horizontal direction, and dy represents the calculation of the gradient sobelY in the vertical direction. The gradient amplitude mag and gradient direction ang are also calculated. The calculation formula is as follows:

[0120] sobelX(x,y)=I(x,y)*kx; sobelY(x,y)=I(x,y)*ky;

[0121]

[0122] ng(x,y)=arctan(sobelY(x,y) / sobelX(x,y));

[0123] Then, the gradient direction is quantized, and the gradient direction ang (unit is radian, -π~π) is quantized into P directions. The formula for gradient direction quantization is:

[0124]

[0125] Among them, the value range of bin_ang is {0,1,2,...,P-1};

[0126] Finally, the gradient direction and amplitude distribution of each rectangular sub-region on the feature image i are calculated, that is, the proportion of the number of pixels in each gradient direction to the total number of pixels is counted and weighted by the gradient amplitude. For the i-th rectangular sub-region, the gradient direction histogram feature is recorded as The calculation formula is as follows:

[0127]

[0128] Among them, bin_ang i (x,y) represents the quantized value of the gradient direction in the i-th rectangular sub-region, and its value range is {0,1,...,P-1}. It represents the probability distribution of the gradient direction and magnitude within the rectangular sub-region, and describes the characteristic information of the grayscale change direction and magnitude.

[0129] In specific application scenarios, LBP is a coding mode that describes the direction of grayscale change in a feature area. In the process of LBP histogram of rectangular sub-areas, the feature image can be LBP-encoded. The rotation-invariant LBP encoding method is used. For the pixel point (x, y) on the feature image I, the 3×3 rectangular neighborhood centered on it is recorded as:

[0130]

[0131] Perform LBP encoding on the pixel point (x, y). The encoding method is as follows: with I(x, y) as the center, compare the 8 intensity values ​​around it with I(x, y) one by one. If it is greater than I(x, y), the corresponding position is marked as 1, otherwise it is marked as 0, and 8 binary digits are obtained. Arrange the 8 binary digits in the order of (x-1, y-1) → (x, y-1) → (x+1, y-1) → (x-1, y) → (x, y) → (x+1, y) → (x-1, y+1) → (x, y+1) → (x+1, y+1) to obtain an 8-bit binary number, called LBP encoding, which is converted into a decimal number in the range of 0 to 255. Taking into account the problem of directional rotation, the binary number is continuously shifted cyclically to obtain a series of LBP feature values. From these LBP code values, a minimum value is selected as the LBP code of the pixel point (x, y), recorded as LBP (x, y); then, the LBP code value is quantized, and the quantization number is recorded as M. The image grayscale value is first normalized to [0, 1] and then quantized to [0, M-1]. For the pixel point (x, y) on the feature image I, the quantization formula of the LBP code is as follows:

[0132]

[0133] Among them, min(LBP) and max(LBP) represent the minimum and maximum values ​​of the LBP encoding value of the feature image respectively;

[0134] Finally, the LBP distribution of each rectangular sub-region on the feature image I is calculated, that is, the ratio of the number of pixels at each LBP coding level to the total number of pixels. For the i-th rectangular sub-region, the LBP histogram feature is recorded as The calculation formula is as follows:

[0135]

[0136] Among them, LBP i quant (x,y) represents the LBP coded quantized value of the pixel in the i-th rectangular area, and its value range is {0,1,...,M-1}. rSize represents the size of the rectangular sub-area, that is, the number of pixels in the rectangular sub-area. Represents the LBP probability distribution within the rectangular sub-region, describing the characteristic information of the grayscale change direction distribution.

[0137] At this point, the feature descriptor can be expressed as:

[0138]

[0139] And the dimension of the feature descriptor is (K+P+M).

[0140] Step S205: For each rectangular sub-region, based on the feature descriptor, calculate the correlation value between the rectangular sub-regions of every two images in the low-level features, select the two images with the largest correlation value as the feature images of the rectangular sub-region, and use the mean value of the feature descriptors of the rectangular sub-region in the feature image as the feature of interest of the rectangular sub-region.

[0141] In practical applications, for each low-level feature, each image in the low-level feature is divided into corresponding rectangular sub-regions. After calculating the feature descriptor of each rectangular sub-region, for each rectangular sub-region, the correlation value between the rectangular sub-regions of every two images in the low-level feature can be calculated based on the feature descriptor. The two images with the largest correlation value are selected as the feature images of the rectangular sub-region, and the average value of the feature descriptors of the rectangular sub-region in the feature image is used as the feature of interest of the rectangular sub-region.

[0142] For easier understanding, see Figure 3 , assuming that the feature vector extracted from the i-th rectangular sub-region on the j-th feature image on the convolutional layer 7 of the VGG16 model is recorded as For the i-th rectangular area, any two feature images I a , I b The correlation between them is denoted as C a,b . And the cosine distance method is used to calculate the correlation between rectangular sub-regions:

[0143]

[0144] in, and Represent the feature image I a , I b The eigenvector on the i-th rectangular area;

[0145] Afterwards, for the i-th rectangular sub-region, the correlation results between the 128 feature images form a correlation sequence C i :

[0146]

[0147] Correlation sequence C iThe maximum value max(C i ) corresponds to the two feature images I m , I n is the target image of the ith rectangular sub-region, then on the ith rectangular sub-region, the feature image I m , I n The extracted feature descriptors is the feature of interest, and the feature vector of interest extracted from the i-th rectangular sub-region is the mean of the two features:

[0148]

[0149] Step S206: For each low-level feature, the interesting features of all rectangular sub-regions of the low-level feature are used as interesting feature vectors.

[0150] Step S207: Train the target model based on the feature vector of interest and the depth feature to obtain a trained target model.

[0151] In practical applications, when training the target model based on the feature vector of interest and the deep features, all the feature vectors of interest can be fused with the deep features to obtain the target features, such as Figure 4 As shown; determine the target type of the target feature; and train the target model based on the target feature and the target type.

[0152] In a specific application scenario, in the process of determining the target type of the target feature, the target feature can be processed based on the DNN classification model to obtain the target type. The DNN classification model in this application can include three fully connected layers and one softmax layer, such as Figure 5 As shown, when the source model is the VGG16 model, the dimensions of the first fully connected layer and the linear transformation layer of the second fully connected layer can be 4096, the activation layer can be a ReLU activation function, and the Dropout rate can be 0.5; the dimensions of the linear transformation layer of the third fully connected layer and the softmax layer are both the number of categories of the image dataset, which can be determined according to actual needs, such as 1000, etc. The activation layer of the third fully connected layer can be a ReLU activation function, and the Dropout rate can be 0.5, etc.

[0153] Step S208: Process the target image based on the target model to obtain a processing result.

[0154] See also Figure 6 , Figure 6 A schematic diagram of the structure of an image processing system provided in an embodiment of the present application.

[0155] An image processing system provided in an embodiment of the present application may include:

[0156] A first acquisition module 101 is used to acquire a target image to be processed;

[0157] The second acquisition module 102 is used to obtain the feature image of the convolution hidden layer of the trained source model;

[0158] A first setting module 103 is used to use the feature image of the last layer as a deep feature and the remaining feature images as low-level features;

[0159] A first extraction module 104 is configured to extract, for each low-level feature, a feature vector of interest of the low-level feature;

[0160] The first training module 105 is used to train the target model based on the feature vector of interest and the depth feature to obtain a trained target model;

[0161] The first processing module 106 is configured to process the target image based on the target model to obtain a processing result.

[0162] In an image processing system provided by an embodiment of the present application, a first extraction module may include:

[0163] A first division unit is used to divide each image in the low-level features into corresponding rectangular sub-regions;

[0164] A first computing unit, configured to compute a feature descriptor for each rectangular sub-region;

[0165] A first determining unit is configured to calculate, for each rectangular sub-region, based on the feature descriptor, a correlation value between the rectangular sub-regions of each two images in the low-level features, select the two images with the largest correlation values ​​as feature images of the rectangular sub-region, and use the average value of the feature descriptors of the rectangular sub-region in the feature images as the feature of interest of the rectangular sub-region;

[0166] The second determining unit is configured to take the interesting features of all rectangular sub-regions as interesting feature vectors.

[0167] An image processing system provided by an embodiment of the present application, the first computing unit can be specifically used to: for each rectangular sub-region, calculate the grayscale histogram, gradient direction histogram and LBP histogram of the rectangular sub-region, and connect the grayscale histogram, gradient direction histogram and LBP histogram to obtain a feature descriptor.

[0168] In an image processing system provided in an embodiment of the present application, a first training module may include:

[0169] The first fusion unit is used to fuse all the feature vectors of interest with the deep features to obtain the target features;

[0170] a third determining unit, configured to determine a target type of the target feature;

[0171] The first training unit is used to train the target model based on target features and target types.

[0172] In an image processing system provided by an embodiment of the present application, the third determination unit can be specifically used to: process target features based on a DNN classification model to obtain a target type.

[0173] An image processing system provided by an embodiment of the present application, wherein the DNN classification model includes three fully connected layers and one softmax layer.

[0174] In an image processing system provided by an embodiment of the present application, a first acquisition module may include:

[0175] A first acquiring unit, configured to acquire an initial image to be processed;

[0176] The first processing unit is configured to perform cropping and / or interpolation and / or brightness normalization processing on the initial image to obtain a target image.

[0177] This application also provides an image processing device and a computer-readable storage medium, both of which have the corresponding effects of the image processing method provided in the embodiment of this application. Figure 7 , Figure 7 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0178] An image processing device provided in an embodiment of the present application includes a memory 201 and a processor 202. The memory 201 stores a computer program. When the processor 202 executes the computer program, the following steps are implemented:

[0179] Obtain the target image to be processed;

[0180] Get the feature image of the convolutional hidden layer of the trained source model;

[0181] The feature image of the last layer is used as the deep feature, and the remaining feature images are used as low-level features;

[0182] For each low-level feature, extract the interesting feature vector of the low-level feature;

[0183] Train the target model based on the feature vector of interest and the deep features to obtain a trained target model;

[0184] The target image is processed based on the target model to obtain a processing result.

[0185] An embodiment of the present application provides an image processing device, including a memory 201 and a processor 202. The memory 201 stores a computer program, and when the processor 202 executes the computer program, the following steps are implemented: each image in the low-level feature is divided into corresponding rectangular sub-regions; a feature descriptor of each rectangular sub-region is calculated; for each rectangular sub-region, based on the feature descriptor, a correlation value between the rectangular sub-regions of every two images in the low-level feature is calculated, the two images with the largest correlation values ​​are selected as feature images of the rectangular sub-region, and the average value of the feature descriptors of the rectangular sub-regions in the feature images is used as a feature of interest in the rectangular sub-region; and the features of interest in all rectangular sub-regions are used as a feature vector of interest.

[0186] An image processing device provided in an embodiment of the present application includes a memory 201 and a processor 202. The memory 201 stores a computer program, and when the processor 202 executes the computer program, it implements the following steps: for each rectangular sub-region, calculate the grayscale histogram, gradient direction histogram, and LBP histogram of the rectangular sub-region, and connect the grayscale histogram, gradient direction histogram, and LBP histogram to obtain a feature descriptor.

[0187] An embodiment of the present application provides an image processing device, including a memory 201 and a processor 202. The memory 201 stores a computer program, and when the processor 202 executes the computer program, it implements the following steps: fusing all feature vectors of interest with depth features to obtain target features; determining the target type of the target features; and training a target model based on the target features and the target type.

[0188] An image processing device provided in an embodiment of the present application includes a memory 201 and a processor 202. The memory 201 stores a computer program. When the processor 202 executes the computer program, the following steps are implemented: target features are processed based on a DNN classification model to obtain a target type.

[0189] An image processing device provided in an embodiment of the present application includes a memory 201 and a processor 202. The memory 201 stores a computer program. When the processor 202 executes the computer program, the following steps are implemented: the DNN classification model includes three fully connected layers and one softmax layer.

[0190] An image processing device provided in an embodiment of the present application includes a memory 201 and a processor 202. The memory 201 stores a computer program, and when the processor 202 executes the computer program, it implements the following steps: obtaining an initial image to be processed; cropping and / or interpolating and / or brightness normalizing the initial image to obtain a target image.

[0191] See also Figure 8 Another image processing device provided in an embodiment of the present application may further include: an input port 203 connected to the processor 202 for transmitting commands inputted from the outside to the processor 202; a display unit 204 connected to the processor 202 for displaying the processing results of the processor 202 to the outside world; and a communication module 205 connected to the processor 202 for enabling communication between the image processing device and the outside world. The display unit 204 may be a display panel, a laser scanning display, etc. The communication method adopted by the communication module 205 includes, but is not limited to, mobile high-definition link technology (HML), universal serial bus (USB), high-definition multimedia interface (HDMI), wireless connection: wireless fidelity technology (WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, and communication technology based on IEEE802.11s.

[0192] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0193] Obtain the target image to be processed;

[0194] Get the feature image of the convolutional hidden layer of the trained source model;

[0195] The feature image of the last layer is used as the deep feature, and the remaining feature images are used as low-level features;

[0196] For each low-level feature, extract the interesting feature vector of the low-level feature;

[0197] Train the target model based on the feature vector of interest and the deep features to obtain a trained target model;

[0198] The target image is processed based on the target model to obtain a processing result.

[0199] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: each image in the low-level feature is divided into corresponding rectangular sub-regions; a feature descriptor of each rectangular sub-region is calculated; for each rectangular sub-region, based on the feature descriptor, a correlation value between the rectangular sub-regions of every two images in the low-level feature is calculated, the two images with the largest correlation value are selected as feature images of the rectangular sub-region, and the average value of the feature descriptors of the rectangular sub-regions in the feature images is used as the feature of interest of the rectangular sub-region; and the features of interest of all rectangular sub-regions are used as the feature vector of interest.

[0200] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: for each rectangular sub-region, the grayscale histogram, gradient direction histogram and LBP histogram of the rectangular sub-region are calculated, and the grayscale histogram, gradient direction histogram and LBP histogram are connected to obtain a feature descriptor.

[0201] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: fusing all feature vectors of interest with deep features to obtain target features; determining the target type of the target features; and training a target model based on the target features and the target type.

[0202] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: target features are processed based on a DNN classification model to obtain a target type.

[0203] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: the DNN classification model includes three fully connected layers and one softmax layer.

[0204] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining an initial image to be processed; cropping and / or interpolating and / or brightness normalizing the initial image to obtain a target image.

[0205] The computer-readable storage medium involved in this application includes random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the technical field.

[0206] For descriptions of the relevant portions of the image processing system, device, and computer-readable storage medium provided in the embodiments of this application, please refer to the detailed descriptions of the corresponding portions of the image processing method provided in the embodiments of this application, and will not be repeated here. In addition, portions of the above-mentioned technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.

[0207] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0208] The above description of the disclosed embodiments will enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that: include: Obtain the target image to be processed; Get the feature image of the convolutional hidden layer of the trained source model; The feature image of the last layer is used as a deep feature, and the remaining feature images are used as low-level features; For each of the low-level features, extracting a feature vector of interest of the low-level feature; Training a target model based on the feature vector of interest and the depth feature to obtain the trained target model; Processing the target image based on the target model to obtain a processing result; The step of extracting the feature vector of interest of the low-level feature includes: Divide each image in the low-level features into corresponding rectangular sub-regions; Calculating a feature descriptor for each of the rectangular sub-regions; For each rectangular sub-region, based on the feature descriptor, calculating the correlation value between the rectangular sub-regions of every two images in the low-level features, selecting the two images with the largest correlation values ​​as the feature images of the rectangular sub-region, and taking the average value of the feature descriptors of the rectangular sub-region in the feature images as the feature of interest of the rectangular sub-region; The features of interest in all the rectangular sub-regions are used as the feature of interest vector.

2. The method according to claim 1, characterized in that The calculating of the feature descriptor of each rectangular sub-region includes: For each rectangular sub-region, a grayscale histogram, a gradient direction histogram, and an LBP histogram of the rectangular sub-region are calculated, and the grayscale histogram, the gradient direction histogram, and the LBP histogram are connected to obtain the feature descriptor.

3. The method according to any one of claims 1 to 2, characterized in that The training of the target model based on the feature vector of interest and the depth feature includes: Fusing all the feature vectors of interest with the depth features to obtain the target features; determining a target type of the target feature; The target model is trained based on the target features and the target type.

4. The method according to claim 3, characterized in that The determining of the target type of the target feature includes: The target features are processed based on a DNN classification model to obtain the target type.

5. The method according to claim 4, characterized in that The DNN classification model includes three fully connected layers and one softmax layer.

6. The method according to claim 1, characterized in that The step of obtaining a target image to be processed includes: Obtain the initial image to be processed; The initial image is cropped and / or interpolated and / or brightness normalized to obtain the target image.

7. An image processing system, characterized in that: include: A first acquisition module is used to acquire a target image to be processed; The second acquisition module is used to obtain the feature image of the convolution hidden layer of the trained source model; A first setting module is used to use the feature image of the last layer as a deep feature and the remaining feature images as low-level features; A first extraction module is configured to extract, for each of the low-level features, a feature vector of interest of the low-level feature; A first training module is used to train the target model based on the feature vector of interest and the depth feature to obtain the trained target model; A first processing module, configured to process the target image based on the target model to obtain a processing result; Wherein, the first extraction module includes: A first division unit is configured to divide each image in the low-level features into corresponding rectangular sub-regions; A first calculation unit, configured to calculate a feature descriptor for each rectangular sub-region; a first determining unit configured to calculate, for each rectangular sub-region, based on the feature descriptor, a correlation value between the rectangular sub-regions of each two images in the low-level features, select the two images with the largest correlation values ​​as feature images of the rectangular sub-region, and use the average value of the feature descriptors of the rectangular sub-region in the feature images as the feature of interest of the rectangular sub-region; The second determining unit is configured to use the features of interest in all the rectangular sub-regions as the feature of interest vector.

8. An image processing device, characterized in that include: memory for storing computer programs; A processor, configured to implement the steps of the image processing method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image processing method according to any one of claims 1 to 6.

Images