Image classification method based on random image mask and double-flow neural network

Abstract

The invention discloses an image classification method based on a random image mask and a double-flow neural network, and the method comprises the steps: obtaining a to-be-classified image, inputting the to-be-classified image into a trained double-flow neural network, and obtaining the category of the to-be-classified image. According to the image classification method based on the random image mask and the double-flow neural network, dual representation of image data is realized through the mask image and the spliced image, and the mask image covers the original image through the mask block, so that the network learning is forced to deduce the overall content from the residual information; according to the spliced image, a finer local feature view is provided by splicing the local information of the original image covered by the mask blocks, the network is helped to capture the fine structure of the image, and meanwhile, the network is trained through the mask image and the spliced image, so that the network learns the overall characteristics only through part of information in the image, and the network learning efficiency is improved. And the generalization ability of the network can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and in particular relates to an image classification method based on random image mask and dual-stream neural network. Background Art

[0002] In the field of computer vision, image processing and analysis technologies are gaining increasing attention. In particular, the rapid development of deep learning has enabled convolutional neural networks (CNNs) to achieve remarkable results in visual image tasks (such as image classification, object detection, etc.).

[0003] However, traditional single-channel input methods often fail to fully utilize the rich information in images when faced with highly diverse and complex datasets. Mask-based image processing methods, as a common data enhancement method, are widely used in tasks such as image classification and target detection. For example, in image classification tasks, by randomly applying masks to images, the model can be forced to learn more robust feature representations using less regional information, thereby improving the generalization performance of the model. At the same time, in target detection tasks, mask-based image processing methods can be used to simulate partial occlusion or disappearance of target objects, helping the model learn to accurately detect and track targets when target information is incomplete. This method helps to improve the generalization performance of the model in complex, real-world scenes, but at the same time, random masking methods may also cause more serious local key information loss and more serious occlusion, resulting in reduced model performance. Summary of the invention

[0004] The purpose of the present invention is to solve the problems raised in the background technology and to propose an image classification method based on random image mask and dual-stream neural network.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention proposes an image classification method based on random image mask and two-stream neural network, comprising obtaining an image to be classified, inputting the image to a trained two-stream neural network, and obtaining a category to which the image to be classified belongs;

[0007] When training the two-stream neural network, an original image is extracted from a training set, a mask block is selected from a preset mask block list, and the mask block is moved from the upper left corner of the original image according to a set step size until the entire original image is traversed;

[0008] The pixel values ​​of the position area where the mask block is moved are set to zero, and then superimposed with the original image to obtain the mask image;

[0009] Splicing the regions of the original image corresponding to all mask blocks in the mask image to obtain a spliced ​​image;

[0010] Perform preprocessing operations on the mask image and the stitched image;

[0011] The preprocessed mask image and the spliced ​​image are used as input data to train the two-stream neural network until the two-stream neural network converges.

[0012] Preferably, the mask block list includes predefined mask blocks of different sizes.

[0013] Preferably, the size of the mask block selected from the mask list is defined as Grid_Size=[G x ,G y ], the default step size is defined as Stride = [S x ,S y ], and satisfy the following relationship:

[0014]

[0015]

[0016] Where W represents the width of the original image, H represents the height of the original image, and G x Indicates the width of the mask block, G y Indicates the height of the mask block, S x Represents the pixel distance along the width direction of the original image in the step length, S y Indicates the pixel distance along the height direction of the original image in the step size.

[0017] Preferably, when the mask block moves on the original image according to the set step size, each pixel point on the mask block is moved by G along the width direction of the original image. x +S x pixels and moves G along the height direction of the original image y +S y Pixel distance.

[0018] Preferably, in the process of stitching the regions of the original image corresponding to all mask blocks in the mask image, when the regions are at the same height, the regions are stitched sequentially along the width direction; when the regions are at the same width, the regions are stitched sequentially along the height direction.

[0019] Preferably, the pretreatment process includes:

[0020] The mask image is randomly scaled, randomly cropped, and randomly horizontally flipped in sequence;

[0021] The spliced ​​image is randomly scaled and randomly cropped in sequence to adjust its size to half the size of the mask image, and then randomly flipped horizontally.

[0022] Preferably, the two-stream neural network includes a first branch and a second branch in parallel, both of which adopt the backbone structure of the CNN model, use the mask image as the input of the first branch, and use the spliced ​​image as the input of the second branch, and then fuse the outputs of the first branch and the second branch to obtain fused features, and finally input the fused features into the remaining structure of the CNN model in sequence to obtain the output of the two-stream neural network.

[0023] Preferably, the first branch and the second branch both adopt the backbone structure of the ResNet-34 network, the first branch includes a first convolutional layer and a second convolutional layer connected in sequence and having a downsampling function, and the second branch includes a first convolutional layer and a second convolutional layer connected in sequence and not having a downsampling function;

[0024] The fusion operation is a splicing operation;

[0025] The fused features are passed through the remaining structure of the ResNet-34 network in sequence to obtain the output of the two-stream neural network.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] This image classification method based on random image masks and two-stream neural networks realizes dual representation of image data through mask images and spliced ​​images. The mask image covers the original image through mask blocks, forcing the network to learn the ability to infer the overall content from the remaining information. The spliced ​​image provides a more detailed local feature view by splicing the local information of the original image covered by the mask block, which helps the network capture the fine structure of the image. At the same time, the network is trained by mask images and spliced ​​images, so that the network learns the overall characteristics only through partial information in the image, which helps to improve the generalization ability of the network, reduce the risk of overfitting, improve the global interaction ability of the network, and improve the accuracy of target recognition, thereby achieving higher classification accuracy and efficiency in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of training a two-stream neural network in an image classification method based on random image masks and a two-stream neural network according to the present invention;

[0029] Figure 2 The two-stream neural network of the present invention predicts partial prediction results of airplanes in the CIFAR-10 classification data set;

[0030] Figure 3The two-stream neural network of the present invention predicts partial prediction results of trucks in the CIFAR-10 classification dataset;

[0031] Figure 4 This is a partial prediction result of the two-stream neural network of the present invention for ships in the CIFAR-10 classification dataset. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] In one embodiment, Figure 1 As shown, an image classification method based on random image mask and two-stream neural network is provided, including:

[0034] Obtain the image to be classified, input it into the trained two-stream neural network, and obtain the category to which the image to be classified belongs; directly input the image to be classified into the trained two-stream neural network to obtain the image category to which it belongs.

[0035] Training a two-stream neural network (where Figure 1 Flow chart of training two-stream neural network):

[0036] Step 1: When training a two-stream neural network, extract the original image from the training set (i.e. Figure 1 The original input image in (in) and its corresponding label information (first, the label information of the original image needs to be set, that is, the classification category of the original image), select a mask block from the preset mask block list, and move the mask block from the upper left corner of the original image according to the set step size until the entire original image is traversed;

[0037] The mask block list includes predefined mask blocks of different sizes. The predefined mask blocks of different sizes are used to improve the diversity of mask block occlusion and the fine granularity of the sampling area. For example, three groups of mask blocks of different sizes are included, namely In each round of training, use the random.choice function to randomly select one of the three groups;

[0038] The size of the mask block selected from the mask list is defined as Grid_Size = [G x ,G y ], the default step size is defined as Stride = [S x ,S y ], and satisfy the following relationship:

[0039]

[0040]

[0041] Where W represents the width of the original image, H represents the height of the original image, and G x Indicates the width of the mask block, G y Indicates the height of the mask block, S x Represents the pixel distance along the width direction of the original image in the step length, S y Indicates the pixel distance along the height direction of the original image in the step size; usually G is selected x With G y equal to maintain the uniformity of the mask; ensure that S x and S y All are greater than 0, so as to maintain a certain blank area in the original image and avoid overlapping of mask blocks;

[0042] When the mask block moves on the original image according to the set step size, each pixel on the mask block is moved by G along the width direction of the original image. x +S x pixels and moves G along the height direction of the original image y +S y pixel distance; that is, starting from the upper left corner of the original image, slide in the width direction every G pixels. x +S x Set a mask block for each pixel in the height direction. y +S y A mask block is set for each pixel until the entire original image is traversed.

[0043] Step 2: Set the pixel value of the position area where the mask block is moved to zero, and then superimpose it with the original image to obtain a mask image;

[0044] First, the area under the mask block coordinates is set to 0 pixels (the pixel value of the mask block is set to 0, not the original image), and the cv2.bitwise_and function is used to superimpose the mask block with the original image to obtain the mask image.

[0045] Step 3: stitching the regions of the original image corresponding to all mask blocks in the mask image to obtain a stitched image (further operations are performed on the basis of the mask image to realize multiplexing of the mask image);

[0046] The specific stitching process is: when the regions are at the same height, the regions (i.e. the regions of the original image corresponding to the mask blocks) are stitched in sequence along the width direction; when the regions (i.e. the regions of the original image corresponding to the mask blocks) are at the same width, the regions are stitched in sequence along the height direction; that is, the regional information extracted in the width direction is stitched along the width direction of the original image, and the regional information extracted in the height direction is stitched along the height direction of the original image, and the resulting image is the stitched image (i.e. Figure 1 The masked stitched image in ).

[0047] Step 4: preprocessing the mask image and the spliced ​​image;

[0048] The preprocessing process includes:

[0049] The mask image is randomly scaled, randomly cropped, and randomly horizontally flipped in sequence;

[0050] The spliced ​​image is randomly scaled and randomly cropped in sequence to adjust its size to half the size of the mask image, and then randomly flipped horizontally.

[0051] For example, the mask image is randomly scaled and randomly cropped to adjust its size to 224×224 pixels, and its data format is converted (the image is converted to tensor format) and image normalization is performed (normalization is performed using mean [0.5, 0.5, 0.5] and standard deviation [0.5, 0.5, 0.5]), and then randomly flipped horizontally to achieve data enhancement;

[0052] The spliced ​​image is randomly scaled and randomly cropped to make it resized to 112×112 pixels, and the data format is converted (the image is converted into a tensor format) and the image is normalized (normalized using the mean [0.5, 0.5, 0.5] and standard deviation [0.5, 0.5, 0.5]), and then randomly horizontally flipped to achieve data enhancement.

[0053] Step 5: Use the preprocessed mask image and the spliced ​​image as input data to train the two-stream neural network until the two-stream neural network converges (calculate the loss through the label information and update the parameters of the two-stream neural network until the loss converges);

[0054] Among them, the two-stream neural network includes a first branch and a second branch in parallel (the structural form of the first branch is consistent with that of the second branch), and the first branch and the second branch both adopt the backbone structure of the CNN model (such as VGG, ResNet) (wherein the backbone structure can be the backbone structure from the input to any structure in the CNN model, and in order to make the sizes of the two fused features consistent before the fusion operation, the structure is set to have a downsampling function or not), the mask image is used as the input of the first branch, and the spliced ​​image is used as the input of the second branch, and then the outputs of the first branch and the second branch are fused to obtain fused features, and finally the fused features are sequentially input into the remaining structures of the CNN model to obtain the output of the two-stream neural network.

[0055] As in this embodiment, the first branch and the second branch both adopt the backbone structure of the ResNet-34 network, the first branch includes a first convolutional layer and a second convolutional layer (from data input to output direction) that are sequentially connected and have a downsampling function, and the second branch includes a first convolutional layer and a second convolutional layer (from data input to output direction) that are sequentially connected and do not have a downsampling function;

[0056] The fusion operation is a concatenation operation;

[0057] The fused features are sequentially passed through the remaining structure of the ResNet-34 network (three consecutive first residual modules, four consecutive second residual modules, six consecutive third residual modules, three consecutive fourth residual modules, an average pooling layer, and a fully connected layer. The first residual module and the fully connected layer do not have the downsampling function, and the rest have the downsampling function) to obtain the output of the two-stream neural network.

[0058] At the end of each round of training, the performance of the model on the validation set is evaluated, and the learning rate and other hyperparameters are dynamically adjusted according to the validation results until the model converges to ensure the accuracy and generalization ability of the model.

[0059] In another embodiment, the present application uses the disclosed CIFAR-10 and CIFAR-100 classification data sets, which are small data sets for identifying natural scene objects compiled by Hinton's students Alex Krizhevsky and Ilya Sutskever, wherein CIFAR-10 contains 10 categories of images, namely airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships and trucks, each category has 6000 color images of size 32x32, while CIFAR-100 has 100 categories, each category has 600 color images of size 32x32. Both data sets contain 60,000 images. In this embodiment, 50,000 images are used as training sets, and another 10,000 images are used as test sets. First, the training sets and their corresponding annotation information in the CIFAR-10 and CIFAR-100 data sets are read. After steps 1 to 5, the overall accuracy (Accuracy) of the model prediction is used as the main evaluation indicator of the detection results on the two data sets. In this embodiment, the ResNet-34 network model is used as an example to design a two-stream neural network (i.e., the structure in step 5), and the performance is judged by comparing the existing models. The results are shown in Table 1 (the accuracy of image classification compared with the network model of the present application and the existing models, and the existing models are ResNet-34 network, MobileNet-V4 network, GhostNet-V3 network and FasterNet-S network). It can be seen from Table 1 that the network structure proposed in the present invention can effectively improve the performance of the model, and the average accuracy of the network model designed based on this method is generally better than the existing methods.

[0060] Table 1. Comparison results of deep neural network models trained with traditional region discarding method

[0061]

[0062] in Figure 2-Figure 3 They are partial recognition results of the network model of this application on CIFRAR-10, where Figure 2 The prediction is airplane with a probability of 100%, Figure 3 The prediction is Truck with a probability of 100%, Figure 4 The prediction is Ship with a probability of 100%.

[0063] This image classification method based on random image masks and two-stream neural networks realizes dual representation of image data through mask images and spliced ​​images. The mask image covers the original image through mask blocks, forcing the network to learn the ability to infer the overall content from the remaining information. The spliced ​​image provides a more detailed local feature view by splicing the local information of the original image covered by the mask block, which helps the network capture the fine structure of the image. At the same time, the network is trained by mask images and spliced ​​images, so that the network learns the overall characteristics only through partial information in the image, which helps to improve the generalization ability of the network, reduce the risk of overfitting, improve the global interaction ability of the network, and improve the accuracy of target recognition, thereby achieving higher classification accuracy and efficiency in practical applications.

[0064] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image classification method based on random image mask and two-stream neural network, characterized in that: The image classification method based on random image mask and two-stream neural network includes: Obtain the image to be classified, input it into the trained two-stream neural network, and obtain the category to which the image to be classified belongs; When training the two-stream neural network, an original image is extracted from a training set, a mask block is selected from a preset mask block list, and the mask block is moved from the upper left corner of the original image according to a set step size until the entire original image is traversed; The pixel values ​​of the position area where the mask block is moved are set to zero, and then superimposed with the original image to obtain the mask image; Splicing the regions of the original image corresponding to all mask blocks in the mask image to obtain a spliced ​​image; Perform preprocessing operations on the mask image and the stitched image; The preprocessed mask image and the spliced ​​image are used as input data to train the two-stream neural network until the two-stream neural network converges.

2. The image classification method based on random image mask and two-stream neural network as claimed in claim 1, characterized in that: The mask block list contains predefined mask blocks of different sizes.

3. The image classification method based on random image mask and two-stream neural network as claimed in claim 1, characterized in that: The size of the mask block selected from the mask list is defined as Grid_Size = [G x ,G y ], the default step size is defined as Stride = [S x ,S y ], and satisfy the following relationship: Where W represents the width of the original image, H represents the height of the original image, and G x Indicates the width of the mask block, G y Indicates the height of the mask block, S x Represents the pixel distance along the width direction of the original image in the step length, S y Indicates the pixel distance along the height direction of the original image in the step size.

4. The image classification method based on random image mask and two-stream neural network as claimed in claim 3, characterized in that: When the mask block moves on the original image according to the set step size, each pixel point on the mask block is moved by G along the width direction of the original image. x +S x pixels and moves G along the height direction of the original image y +S y Pixel distance.

5. The image classification method based on random image mask and two-stream neural network as claimed in claim 1, characterized in that: In the process of splicing the regions of the original image corresponding to all mask blocks in the mask image, when the regions are at the same height, the regions are sequentially spliced ​​along the width direction; when the regions are at the same width, the regions are sequentially spliced ​​along the height direction.

6. The image classification method based on random image mask and two-stream neural network as claimed in claim 1, characterized in that: The pre-treatment process includes: The mask image is randomly scaled, randomly cropped, and randomly horizontally flipped in sequence; The spliced ​​image is randomly scaled and randomly cropped in sequence to adjust its size to half the size of the mask image, and then randomly flipped horizontally.

7. The image classification method based on random image mask and two-stream neural network as claimed in claim 1, characterized in that: The two-stream neural network includes a first branch and a second branch in parallel. The first branch and the second branch both adopt the backbone structure of the CNN model. The mask image is used as the input of the first branch, and the spliced ​​image is used as the input of the second branch. The outputs of the first branch and the second branch are then fused to obtain fused features. Finally, the fused features are sequentially input into the remaining structure of the CNN model to obtain the output of the two-stream neural network.

8. The image classification method based on random image mask and two-stream neural network as claimed in claim 7, characterized in that: The first branch and the second branch both adopt the backbone structure of the ResNet-34 network, the first branch includes a first convolutional layer and a second convolutional layer that are sequentially connected and have a downsampling function, and the second branch includes a first convolutional layer and a second convolutional layer that are sequentially connected and do not have a downsampling function; The fusion operation is a splicing operation; The fused features are passed through the remaining structure of the ResNet-34 network in sequence to obtain the output of the two-stream neural network.

Images