An image classification method, device, storage medium and electronic equipment

Abstract

The image classification method provided in the specification adopts a pre-trained image classification model to classify a to-be-processed image. After an image feature extraction subnetwork in the model extracts an image feature of the to-be-processed image, the image feature of the to-be-processed image is split into multiple split image features, each split image feature is simultaneously processed by multiple parallel feature processing subnetworks in the model, finally each processed split image feature is fused by a fusion subnetwork in the model to obtain a fused image feature, and the to-be-processed image is classified according to the fused image feature. The above method adopts a splitting and parallel processing mode for the to-be-processed image feature, effectively reducing the time required for processing the to-be-processed image feature, and improving the efficiency of the image classification model in image classification.

Description

TECHNICAL FIELD

[0001] The present specification relates to the field of image processing, and in particular, to an image classification method and device, a storage medium, and an electronic device. BACKGROUND

[0002] As one of the important technologies in the field of image processing, image classification technology has a very wide range of applications due to its simple and direct but highly versatile functions. For example, face recognition, object detection, scene classification, etc. Therefore, people's expectations for the effects achieved by image classification technology are also getting higher and higher. In order to make image classification technology have better classification effect, researchers have been studying image classification technology.

[0003] At present, most image classification technologies are implemented through various neural network models based on convolutional neural networks (CNN). In recent years, better classification models have been proposed, such as MobileNet series, ShuffleNet series, etc. Each new model for image classification is proposed, which has higher accuracy. On the other hand, by comparing the data and indicators disclosed by these neural networks with high accuracy, it can be found that the classification efficiency of these neural networks is relatively slow.

[0004] Obviously, in today's fast-paced lifestyle, users also have high requirements for classification efficiency. Therefore, for neural networks used for image classification, the importance of classification efficiency is no less than the accuracy of classification. The accuracy of image classification is certainly important, but it is not a good strategy to excessively sacrifice classification efficiency for higher classification accuracy. The image classification model in the prior art has high accuracy, but the image classification efficiency is slow. In other words, the image classification model in the prior art cannot balance the classification accuracy and the classification efficiency. SUMMARY

[0005] The present specification provides an image classification method, device, storage medium and electronic device to partially solve the above problems existing in the prior art.

[0006] The present specification adopts the following technical solutions:

[0007] The present specification provides an image classification method, comprising:

[0008] obtaining a to-be-processed image;

[0009] inputting the to-be-processed image into an image feature extraction subnetwork in a pre-trained image classification model to extract to-be-processed image features of the to-be-processed image through the image feature extraction subnetwork;

[0010] splitting the image feature to be processed to obtain at least two split image features;

[0011] inputting each split image feature into each feature processing subnetwork in the image classification model respectively, so as to process the respective input split image features through the feature processing subnetworks in parallel;

[0012] inputting each split image feature processed through the feature processing subnetwork into a feature fusion subnetwork in the image classification model, so as to fuse the input split image features through the feature fusion subnetwork to obtain a fused image feature;

[0013] classifying the image to be processed according to the fused image feature.

[0014] Optionally, the image feature extraction subnetwork comprises an extraction layer, a first processing layer.

[0015] extracting the image feature to be processed of the image to be processed through the image feature extraction subnetwork, specifically comprising:

[0016] extracting an initial image feature of the image to be processed through the extraction layer;

[0017] reducing the channel number of the initial image feature through the first processing layer to obtain the image feature to be processed.

[0018] Optionally, the feature fusion subnetwork comprises a splicing layer, a fusion layer, and a second processing layer.

[0019] fusing the input split image features through the feature fusion subnetwork to obtain a fused image feature, specifically comprising:

[0020] splicing the split image features processed through the feature processing subnetworks to obtain spliced image features through the splicing layer;

[0021] increasing the channel number of the spliced image features through the second processing layer to obtain expanded spliced image features;

[0022] fusing the expanded spliced image features and the initial image feature through the fusion layer to obtain the fused image feature.

[0023] Optionally, each feature processing subnetwork comprises a plurality of convolution layers.

[0024] processing the respective input split image features through the feature processing subnetworks in parallel, specifically comprising:

[0025] For each feature processing subnetwork, the split image features inputting the feature processing subnetwork are processed through each convolutional layer in the feature processing subnetwork in sequence, wherein the sum of the channel numbers of the split image features processed by the feature processing subnetworks is the same as the channel number of the image feature to be processed.

[0026] Optionally, the split image features inputting the feature processing subnetwork are processed through each convolutional layer in the feature processing subnetwork in sequence, and specifically comprising:

[0027] For each convolutional layer in the feature processing subnetwork, the residual convolutional kernel is fused with the standard convolutional kernel corresponding to the convolutional layer to obtain the fusion convolutional kernel corresponding to the convolutional layer.

[0028] The split image features inputting the convolutional layer are convolved through the fusion convolutional kernel corresponding to the convolutional layer.

[0029] Optionally, one feature processing subnetwork comprises an ordinary convolutional layer and a down-sampling convolutional layer.

[0030] The split image features inputting the feature processing subnetwork are processed through each convolutional layer in the feature processing subnetwork in sequence, and specifically comprising:

[0031] The split image features inputting the feature processing subnetwork are convolved through the ordinary convolutional layer in the processing subnetwork, and the size and the channel number of the split image features inputting the feature processing subnetwork are reduced through the down-sampling convolutional layer in the feature processing subnetwork.

[0032] Optionally, the feature fusion subnetwork comprises a splicing layer, a fusion layer, and a third processing layer.

[0033] The input split image features are fused through the feature fusion subnetwork to obtain a fusion image feature, and specifically comprising:

[0034] The split image features processed by the feature processing subnetworks are spliced through the splicing layer to obtain a spliced image feature.

[0035] The initial image feature is processed through the third processing layer to reduce the size and the channel number of the initial image feature to obtain a down-sampled image feature, wherein the initial image feature is the initial image feature extracted by the extraction layer in the image feature extraction subnetwork, and the size and the channel number of the down-sampled image feature are the same as the size and the channel number of the spliced image feature.

[0036] The spliced image feature and the down-sampled image feature are fused through the fusion layer to obtain a fusion image feature.

[0037] The specification provides an image classification device, which comprises:

[0038] An acquisition module acquires an image to be processed.

[0039] A feature extraction module inputs the image to be processed into an image feature extraction subnetwork in a pre-trained image classification model to extract image features of the image to be processed through the image feature extraction subnetwork.

[0040] A splitting module splits the image features to be processed to obtain at least two split image features.

[0041] A feature processing module inputs each split image feature into each feature processing subnetwork in the image classification model to process the input split image feature through each feature processing subnetwork in parallel.

[0042] A fusion module inputs each split image feature processed through the feature processing subnetwork into a feature fusion subnetwork in the image classification model to fuse the input split image features through the feature fusion subnetwork to obtain fused image features.

[0043] A classification module classifies the image to be processed according to the fused image features.

[0044] The specification provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above image classification method.

[0045] The specification provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the above image classification method when executing the program.

[0046] The above at least one technical solution adopted by the specification can achieve the following beneficial effects:

[0047] The image classification method provided by the specification classifies the image to be processed using a pre-trained image classification model. After extracting the image features of the image to be processed using the image feature extraction subnetwork in the model, the image features of the image to be processed are split into multiple split image features. Each split image feature is processed through multiple parallel feature processing subnetworks in the model at the same time. Finally, each processed split image feature is fused through the fusion subnetwork in the model to obtain fused image features, and the image to be processed is classified according to the fused image features. The above method splits and processes the image features to be processed in parallel, effectively reducing the time required to process the image features to be processed and improving the efficiency of the image classification model in image classification. BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0049] Figure 1 This is a flowchart illustrating one image classification method described in this specification.

[0050] Figure 2 This is a schematic diagram of the image classification model used in this specification;

[0051] Figure 3 This is a schematic diagram of one structure of the feature processing subnet in this specification;

[0052] Figure 4 This is a schematic diagram of one structure of the image feature extraction subnetwork in this specification;

[0053] Figure 5 This is a schematic diagram of one structure of the feature fusion subnet in this specification;

[0054] Figure 6 This is a schematic diagram of the feature processing subnet used for image downsampling in this specification;

[0055] Figure 7 This is a schematic diagram of a feature fusion subnet structure used in image downsampling processing in this specification;

[0056] Figure 8 This is a schematic diagram illustrating the fusion of residual convolution kernels and standard convolution kernels in this specification.

[0057] Figure 9 This is a schematic diagram of an image classification device provided in this specification;

[0058] Figure 10 The corresponding information provided in this specification Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0059] Image classification technology, as one of the core technologies in the field of image processing, has seen continuous breakthroughs in recent years with the emergence of new neural networks. Taking neural networks for mobile devices as an example, such as Google's MobileNet series and Megvii's ShuffleNet series, these networks have further explored the performance limits of neural networks for image classification. However, in these neural networks, the performance breakthroughs are mostly reflected in the accuracy of image classification, with minimal improvement in image classification efficiency.

[0060] Most of the existing high-performance neural networks are based on residual networks, and have additional residual structures in the network. At the same time, in order to strengthen the transmission of features between each layer, most of these high-performance networks adopt dense connection. Although these methods can give more accurate results when the neural network classifies images, they also cause the neural network to process a large amount of additional information, require a large amount of additional memory access, and ultimately result in low efficiency of the neural network for image classification.

[0061] It can be seen that although the existing image classification technology method has high accuracy, it sacrifices the efficiency of image classification and cannot balance the efficiency and accuracy of image classification. In order to solve this technical problem, the present specification proposes an image classification method, which can greatly improve the efficiency of image classification while ensuring relatively high image classification accuracy, and better balances the efficiency and accuracy of image classification.

[0062] In order to make the purpose, technical scheme and advantages of the present specification clearer, the technical scheme of the present specification will be described clearly and completely in the following combined with specific embodiments of the present specification and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present specification, not all embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0063] In order to make the purpose, technical scheme and advantages of the present specification clearer, the technical scheme of the present specification will be described clearly and completely in the following combined with specific embodiments of the present specification and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present specification, not all embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0064] The technical scheme provided by each embodiment of the present specification will be described in detail below in combination with the drawings.

[0065] Figure 1 The flowchart of the image classification method in the present specification specifically includes the following steps:

[0066] S100: Obtain an image to be processed.

[0067] All steps in the image classification method provided by the present specification can be executed by any electronic device with computing function, such as terminal, server and the like.

[0068] For any technology in the field of image processing, the image to be processed is the basis, and the image classification technology is no exception. As the object of image classification, the image to be processed needs to be obtained first to ensure the subsequent steps.

[0069] S102: input the image to be processed into an image feature extraction subnetwork in a pre-trained image classification model to extract the image feature to be processed of the image to be processed through the image feature extraction subnetwork.

[0070] The image classification method provided in the specification adopts a pre-trained image classification model to complete part of the work in the image classification task, and the structure diagram of the image classification model is as shown in Figure 2 The image classification model includes an image feature extraction subnetwork, at least two feature processing subnetworks, and a feature fusion subnetwork. The image to be processed obtained in step S100 is input into the image classification model, and the image feature to be processed of the image to be processed is extracted through the image feature extraction subnetwork in the image classification model for subsequent steps.

[0071] It should be noted that the image classification model provided in the specification can be applied to any network based on a convolutional neural network, including but not limited to a convolutional neural network, a residual neural network, etc.

[0072] S104: split the image feature to be processed to obtain at least two split image features.

[0073] Generally, the performance of the currently widely used electronic device is often excessive when only processing one image feature. Even if multiple image features are processed at the same time, the processing speed of the electronic device for each image feature will not decrease much compared to when the image feature is processed alone. Therefore, the image feature to be processed obtained in step S102 can be split into multiple split image features without affecting the processing speed of the electronic device too much, and each split image feature can be processed in parallel in the subsequent steps, thereby speeding up the overall efficiency.

[0074] It should be noted that generally, the image feature has three dimensions of channels, width, and height, and the width and height can be referred to as size together. The splitting of the image feature to be processed is in the channel dimension of the image feature to be processed, and the splitting method can be set according to requirements, and the number of channels of each split image feature can be the same or different.

[0075] S106: input each split image feature into each feature processing subnetwork in the image classification model to process the input split image feature in parallel through each feature processing subnetwork.

[0076] As mentioned in step S104, processing each split image feature in a parallel manner can speed up the processing efficiency. Therefore, each feature processing subnetwork can be used to process one split image feature, and all split image features can be processed at the same time to maximize the processing efficiency. The manner in which each feature processing subnetwork processes a split image feature can be set according to the split image feature input into the feature processing subnetwork, and the internal structure of each feature processing subnetwork can be the same or different, which is not limited in the specification.

[0077] S108: input each split image feature processed by the feature processing subnetwork into a feature fusion subnetwork in the image classification model, so as to fuse each input split image feature by the feature fusion subnetwork to obtain a fused image feature.

[0078] The splitting of the to-be-processed image feature separates the complete image feature information onto each split image feature, and each split image feature can only reflect part of the image feature information. Therefore, each split image feature processed by the feature processing subnetwork can be fused together by the feature fusion subnetwork in the image classification model to obtain a fused image feature that can completely reflect all the image feature information.

[0079] When fusing the split image features, there can be various fusion methods, for example, all split image features can be directly spliced together, or different weights can be assigned to each split image feature according to requirements before splicing, which is not limited in the specification.

[0080] S110: classify the to-be-processed image according to the fused image feature.

[0081] After obtaining the fused image feature, the to-be-processed image can be classified according to the fused image feature to complete the image classification task.

[0082] As can be seen from the above steps, the image classification method provided by the specification splits the to-be-processed image feature after extracting the to-be-processed image feature corresponding to the to-be-processed image by the image feature extraction subnetwork to obtain at least two split image features, processes the split image features by the feature processing subnetwork in a parallel manner, and finally fuses the split image features together by the feature fusion subnetwork to obtain a fused image feature that can reflect complete image feature information, which effectively improves the efficiency of image classification on the basis of ensuring the accuracy of image classification.

[0083] The image classification method provided in the specification can be used in a variety of different scenarios, for example, in the aspect of face recognition, in addition to the basic judgment of the corresponding person information according to the face image, it can also be used to quickly detect whether the target person wears a mask or whether the target person is in the black / white list; in addition, it also has a wide range of applications in object recognition, such as detecting whether an object is a specified category of object or classifying an object. The functions in such application scenarios will be repeatedly used in daily life, so the efficiency of executing such functions is very critical, and improving the implementation efficiency of such functions can save a lot of time for users. The image classification method provided in the specification can quickly and effectively implement such functions.

[0084] In some application scenarios, higher image classification accuracy may be required. Therefore, in step S108, when the features of the input split images are fused by the feature fusion subnetwork, the original features of the image to be processed obtained in step S102, i.e., the image-to-be-processed features, can also be fused in to improve the accuracy. Specifically, the split image features processed by each feature processing subnetwork can be spliced to obtain spliced image features, and the spliced image features and the image-to-be-processed features can be fused to obtain fused image features.

[0085] Figure 3 is a schematic diagram of the internal structure of each feature processing subnetwork in the image classification model. Each feature processing subnetwork includes a plurality of convolutional layers, and the specific number X of convolutional layers can be set according to requirements. After the split image features are input into the feature processing subnetwork, they will be processed by each convolutional layer in the feature processing subnetwork in turn. At the same time, when processing the split image features, each feature processing subnetwork makes the sum of the channel numbers of the split image features processed by each feature processing subnetwork equal to the channel number of the image-to-be-processed features, so as to ensure that the channel number of the spliced image features obtained after splicing the split image features is equal to the channel number of the image-to-be-processed features.

[0086] Since the sum of the channel numbers of the split image features processed by each feature processing subnetwork is equal to the channel number of the image-to-be-processed features, the spliced image features and the image-to-be-processed features can maximize the preservation of feature information when they are fused, and the fusion effect is good. For the same purpose, when the spliced image features and the image-to-be-processed features are fused, a superposition method can be used for fusion. The above method can further improve the accuracy of image classification, and almost does not affect the efficiency of image classification.

[0087] On this basis, when processing image features, the amount of information to be processed can be reduced by reducing the channel number of the image features, thereby further improving the efficiency of image classification. At this time, the structure of the image feature extraction subnetwork can be as follows:Figure 4 As shown in the figure, the image feature extraction subnetwork includes an extraction layer and a first processing layer. The initial image features of the image to be processed are extracted through the extraction layer, and the channel number of the initial image features is reduced through the first processing layer to obtain image features to be processed. When the image features to be processed are processed subsequently, the image classification efficiency can be improved in different magnitudes according to the different reduction amounts of the channel number.

[0088] The method for reducing the channel number of the initial image features can be various, for example, the channel number of the initial image features can be reduced through convolution by adjusting the convolution kernel. Specifically, M represents the channel number of the initial image features, N represents the channel number of the image features to be processed, H represents the height of the initial image features and the image features to be processed, and W represents the width of the initial image features and the image features to be processed, where M>N. At this time, the initial image features can be represented as MxHxW, and the image features to be processed can be represented as NxHxW. N adjusted convolution kernels are used to convolve the initial image features to obtain N single-channel image features. The N single-channel image features are spliced to obtain the image features to be processed. The channel number of the adjusted convolution kernel is M, the height is 1, and the width is 1. The channel number of the single-channel image feature is 1, the height is H, and the width is W. The channel number of the image features can be reduced through convolution by using this method, where M, N, H, and W can be set to any value as required, for example, N can be half of M, that is, N=M / 2. In addition, the channel number of the initial image features can also be reduced by directly discarding part of the data in the channel or by weighting and merging part of the channels into one channel. This will not be described in detail herein.

[0089] However, at this time, the original features of the image to be processed are no longer the image features to be processed, but the initial image features extracted by the extraction layer. The channel number of the spliced image features obtained by splicing the split image features processed by each feature processing subnetwork is the same as that of the image features to be processed, which is less than that of the initial image features. Therefore, in order to effectively fuse the initial image features, the spliced image features need to be processed.

[0090] Specifically, the structure of the feature fusion subnetwork can be as shown in the figure. Figure 5 As shown in the figure, the feature fusion subnetwork includes a splicing layer, a fusion layer, and a second processing layer. The split image features processed by each feature processing subnetwork are spliced through the splicing layer to obtain spliced image features. The channel number of the spliced image features is increased through the second processing layer to obtain expanded spliced image features. The expanded spliced image features and the initial image features are fused through the fusion layer to obtain fused image features.

[0091] There are various methods to increase the number of channels in the stitched image features. However, to ensure the best results during subsequent fusion, the method for increasing the number of channels in the stitched image features should correspond to the method for decreasing the number of channels in the initial image features. For example, if the method of adjusting the convolution kernel was used to decrease the number of channels in the initial image features, then the stitched image features can also be convolved using the adjusted convolution kernel to increase the number of channels in the stitched image features. Specifically, N can represent the number of channels in the stitched image features, M can represent the number of channels in the expanded stitched image features, H can represent the height of the stitched image features and the expanded stitched image features, and W can represent the width of the stitched image features and the expanded stitched image features, where N < M. In this case, the stitched image features can be represented as N×H×W, and the expanded stitched image features can be represented as M×H×W. The stitched image features are convolved using M adjusted convolution kernels to obtain M single-channel image features; these M single-channel image features are then stitched together to obtain expanded stitched image features. The adjusted convolution kernels have N channels, a height of 1, and a width of 1; each single-channel image feature has 1 channel, a height of H, and a width of W. Corresponding to the method for reducing initial image features described above, N can still be half of M, i.e., N = M / 2.

[0092] Before fusing with the initial image features, the second processing layer within the feature fusion subnetwork increases the number of channels in the stitched image features to the same number as the initial image features, resulting in expanded stitched image features. These expanded stitched image features can then be fused with the initial image features more effectively. This method accelerates the image classification model's processing speed, further improving image classification efficiency.

[0093] Furthermore, when the image to be processed is large, the size and number of channels of its corresponding image features may also be large, making processing more complex. In this case, the image features to be processed can be reduced by downsampling, that is, the size and number of channels of the image can be reduced, thereby reducing the amount of information to be processed, thus reducing the time required to process image features and speeding up image classification efficiency.

[0094] Available as Figure 4 The image feature extraction subnetwork shown extracts features from the image to be processed, obtaining initial image features and image features with reduced channel count. The structure of each feature processing subnetwork can then be as follows: Figure 6As shown, one feature processing subnetwork includes a normal convolution layer and a down-sampling convolution layer. Generally, there is only one down-sampling convolution layer, and the number Y of normal convolution layers can be set according to requirements. The normal convolution layer in the feature processing subnetwork is used to convolve the split image features input into the feature processing subnetwork, and the down-sampling convolution layer in the feature processing subnetwork is used to reduce the size and the number of channels of the split image features input into the feature processing subnetwork.

[0095] The method of reducing the size and the number of channels of the split image features input into the feature processing subnetwork through the down-sampling convolution layer can be based on the method of reducing the number of channels of image features through convolution by using an adjusted convolution kernel mentioned above, and the size of the adjusted convolution kernel is increased or the step of convolution is expanded, so as to reduce the size and the number of channels of the split image features at the same time. For example, assuming that the number of channels of the split image features before being processed by the down-sampling convolution layer is P, the height is H, the width is W, the number of channels of the split image features after being processed by the down-sampling convolution layer is Q, the height is H / 2, and the width is W / 2, where P>Q, Q adjusted convolution kernels can be used to convolve the split image features before being processed by the down-sampling convolution layer with a step of 2, so as to obtain Q single-channel image features; the Q single-channel image features are spliced, so as to obtain the split image features after being processed by the down-sampling convolution layer; where the number of channels of the adjusted convolution kernel is P, the height is 2, and the width is 2; the number of channels of the single-channel image features is 1, the height is H / 2, and the width is W / 2. Similarly, the values of P, Q, H, and W can be set to any values under the condition of meeting requirements, for example, Q can be half of P, i.e., Q=P / 2.

[0096] It is worth mentioning that, Figure 6 The structure of the feature processing subnetwork given in the above embodiment is that the down-sampling convolution layer is in front of the normal convolution layer, but in actual applications, the order of the normal convolution layer and the down-sampling convolution layer can be freely set, and the structures inside each feature processing subnetwork can also be different.

[0097] Correspondingly, the structure of the feature fusion subnetwork at this time can be as shown in Figure 7 At this time, the feature fusion subnetwork includes a splicing layer, a fusion layer, and a third processing layer. The splicing layer is used to splice the split image features processed by the feature processing subnetworks, so as to obtain a spliced image feature; the third processing layer is used to process an initial image feature, so as to reduce the size and the number of channels of the initial image feature, and obtain a down-sampled image feature, where the initial image feature is an initial image feature extracted by the extraction layer in the image feature extraction subnetwork, and the size and the number of channels of the down-sampled image feature are the same as those of the spliced image feature; the spliced image feature and the down-sampled image feature are fused by the fusion layer, so as to obtain a fused image feature.

[0098] At this point, the size and number of channels of the stitched image features are different from those of the initial image features. Therefore, it is also necessary to downsample the initial image features. That is, the size and number of channels of the initial image features are reduced through the third processing layer to obtain the downsampled image features. When reducing the size and number of channels of the initial image features, either the convolution method mentioned above by adjusting the convolution kernel can be used, or pooling can be used. This specification does not impose any restrictions here.

[0099] To prevent performance degradation, most image classification models extract residual information before processing the images. Similarly, this model can be based on a residual network. However, extracting residual information significantly slows down image classification. Therefore, to address this issue, the residual convolutional kernels used for residual information extraction can be directly fused into the feature processing sub-network, eliminating the need for additional steps to obtain residual information. Specifically, for each convolutional layer in the feature processing sub-network, the residual convolutional kernel is fused with the corresponding standard convolutional kernel to obtain a fused convolutional kernel. This fused convolutional kernel is then used to convolve the segmented image features input to that convolutional layer. In this case, the segmented image features input to that convolutional layer will now carry additional residual information after convolution with the fused convolutional kernel.

[0100] Among them, the fusion operation can be as follows: Figure 8 As shown, assuming that the original standard convolutional kernel in each processing layer of the feature processing subnetwork is 3×3 in size and the residual convolutional kernel used to extract residual information is 1×1 in size, then the residual convolutional kernel can be regarded as a 3×3 convolutional kernel with weights only in the middle and weights of 0 in the other 8 positions. Therefore, the residual convolutional kernel and the standard convolutional kernel can be directly superimposed to obtain the fused convolutional kernel.

[0101] The image classification model used in the image classification method provided in the specification can be trained in various ways, such as supervised learning and reinforcement learning, and can be trained in different content for different application scenarios. For example, in the application scenario of face recognition, the sample face image and the corresponding label can be input into the model as training data for learning, the model can determine the person information corresponding to the face image, and compare the determination result of the model with the label in the training data, and the difference between the determination result of the model and the label in the training data is minimized as the optimization target, and the parameters in the model are adjusted until the accuracy of the model meets the requirements. For another example, in the application scenario of object recognition, different kinds of objects can be input into the model as training data for learning, and the model can be used to classify the objects, and the classification of the model can be fed back through the loss function, and the internal parameters of the model are adjusted until the accuracy of the model meets the requirements.

[0102] The above is the image classification method provided in the specification, based on the same idea, the specification also provides a corresponding image classification device, as shown in Figure 9

[0103] Figure 9 The image classification device provided in the specification is shown in the figure, which specifically comprises:

[0104] The acquisition module 200 acquires the image to be processed;

[0105] The feature extraction module 202 inputs the image to be processed into the image feature extraction subnetwork in the pre-trained image classification model, so as to extract the image feature to be processed of the image to be processed through the image feature extraction subnetwork;

[0106] The splitting module 204 splits the image feature to be processed to obtain at least two split image features;

[0107] The feature processing module 206 inputs each split image feature into each feature processing subnetwork in the image classification model respectively, so as to process the split image feature input by each feature processing subnetwork in parallel;

[0108] The fusion module 208 inputs each split image feature processed by the feature processing subnetwork into the feature fusion subnetwork in the image classification model, so as to fuse the input split image features through the feature fusion subnetwork to obtain a fused image feature;

[0109] The classification module 210 classifies the image to be processed according to the fused image feature.

[0110] In an optional embodiment:

[0111] ​The image feature extraction subnetwork comprises an extraction layer and a first processing layer;

[0112] The feature extraction module 202 is specifically configured to extract initial image features of the image to be processed through the extraction layer, and reduce the number of channels of the initial image features through the first processing layer to obtain image features to be processed.

[0113] In an optional embodiment, the feature fusion subnetwork comprises a splicing layer, a fusion layer and a second processing layer.

[0114] The fusion module 208 is specifically configured to splice the split image features processed by the feature processing subnetworks to obtain spliced image features through the splicing layer, increase the number of channels of the spliced image features through the second processing layer to obtain expanded spliced image features, and fuse the expanded spliced image features and the initial image features through the fusion layer to obtain fused image features.

[0115] In an optional embodiment:

[0116] Each feature processing subnetwork comprises a plurality of convolution layers.

[0117] The feature processing module 206 is specifically configured to, for each feature processing subnetwork, sequentially process the split image features input into the feature processing subnetwork through each convolution layer in the feature processing subnetwork, wherein the sum of the number of channels of the split image features processed by the feature processing subnetworks is the same as the number of channels of the image features to be processed.

[0118] In an optional embodiment:

[0119] The feature processing module 206 is specifically configured to, for each convolution layer in the feature processing subnetwork, fuse a residual convolution kernel and a standard convolution kernel corresponding to the convolution layer to obtain a fused convolution kernel corresponding to the convolution layer, and perform convolution on the split image features input into the convolution layer through the fused convolution kernel corresponding to the convolution layer.

[0120] In an optional embodiment:

[0121] One feature processing subnetwork comprises a normal convolution layer and a down-sampling convolution layer.

[0122] The feature processing module 206 is specifically configured to perform convolution on the split image features input into the feature processing subnetwork through the normal convolution layer in the feature processing subnetwork, and reduce the size and the number of channels of the split image features input into the feature processing subnetwork through the down-sampling convolution layer in the feature processing subnetwork.

[0123] In an optional embodiment:

[0124] The feature fusion subnetwork comprises a splicing layer, a fusion layer and a third processing layer.

[0125] The feature fusion module 208 is specifically configured to splice the split image features processed by the feature processing subnetworks to obtain spliced image features through the splicing layer; process the initial image features to reduce the size and the number of channels of the initial image features to obtain down-sampled image features through the third processing layer, wherein the initial image features are initial image features extracted by an extraction layer in the image feature extraction subnetwork, and the size and the number of channels of the down-sampled image features are the same as those of the spliced image features; and fuse the spliced image features and the down-sampled image features through the fusion layer to obtain fused image features.

[0126] The present specification also provides a computer readable storage medium storing a computer program, and the computer program can be used to execute the above Figure 1 The image classification method is provided.

[0127] The present specification also provides a computer readable storage medium storing a computer program, and the computer program can be used to execute the above Figure 10 The schematic structural diagram of the electronic device is shown. As Figure 10 At the hardware level, the unmanned device comprises a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course, other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to realize the above Figure 1 The image classification method is provided. Of course, in addition to the software implementation, the present specification does not exclude other implementation manners, such as logic devices or the combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to the logic units, but can also be hardware or logic devices.

[0128] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by asking a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code to be compiled is written in a specific programming language, which is called a hardware description language (HDL), and there are many such languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0129] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0130] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0131] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware in implementing the present specification.

[0132] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0133] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0134] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0135] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. ​ one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0136] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0137] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory. The memory is an example of computer-readable media.

[0138] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0139] It should also be noted that the terms "comprising", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0140] Those skilled in the art will appreciate that embodiments of the present specification can be provided as methods, systems or computer program products. Therefore, the present specification can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] The present specification can be described in the general context of computer-executable instructions, such as program modules, executed by computers. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media, including storage devices.

[0142] The various embodiments described in this specification are described using a numbering of embodiments approach: these are each individually integrated contributions pertaining to different but related aspects of the description. Each of the various embodiments can stand on its own, and each can be combined with the subject matter of other embodiments to produce further embodiments. Where appropriate, therefore, the contents of the specification can be regarded as incorporating text of the detailed description under the heading "Embodiments."

[0143] The above description is embodied only by the embodiments of the specification, and is not intended to limit the specification. The specification can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the specification shall be included in the scope of the claims of the present application.

Claims

1. An image classification method, characterized in that, include: Obtain the image to be processed; The image to be processed is input into the image feature extraction subnetwork of a pre-trained image classification model to extract the image features to be processed from the image to be processed through the image feature extraction subnetwork; The features of the image to be processed are split to obtain at least two split image features; Each split image feature is input into each feature processing subnet in the image classification model, so that the input split image features are processed in parallel through each feature processing subnet. Each split image feature processed by the feature processing subnetwork is input into the feature fusion subnetwork in the image classification model, so as to fuse the input split image features through the feature fusion subnetwork to obtain fused image features; Based on the fused image features, classify the image to be processed; Each feature processing subnetwork includes several convolutional layers; The features of each input split image are processed in parallel through a feature processing subnetwork, specifically including: For each feature processing subnetwork, the segmented image features input to the feature processing subnetwork are processed sequentially through each convolutional layer in the feature processing subnetwork. The sum of the number of channels of each segmented image feature after processing by each feature processing subnetwork is the same as the number of channels of the image feature to be processed.

2. The method as described in claim 1, characterized in that, The image feature extraction subnetwork includes: an extraction layer and a first processing layer; The image feature extraction subnetwork extracts the image features to be processed from the image to be processed, specifically including: The initial image features of the image to be processed are extracted through the extraction layer; The number of channels of the initial image features is reduced by the first processing layer to obtain the image features to be processed.

3. The method as described in claim 2, characterized in that, The feature fusion subnet includes: a splicing layer, a fusion layer, and a second processing layer; The feature fusion subnetwork fuses the input split image features to obtain fused image features. Specifically, the stitching layer stitches together the split image features processed by each feature processing subnetwork to obtain stitched image features. The number of channels in the stitched image features is increased by the second processing layer to obtain expanded stitched image features; The expanded and stitched image features are fused with the initial image features through the fusion layer to obtain fused image features.

4. The method as described in claim 1, characterized in that, Each convolutional layer in this feature processing subnetwork sequentially processes the split image features input to the subnetwork, specifically including: For each convolutional layer in this feature processing subnetwork, the residual convolutional kernel is fused with the standard convolutional kernel corresponding to that convolutional layer to obtain the fused convolutional kernel corresponding to that convolutional layer; The split image features input to the convolutional layer are convolved using the fusion convolutional kernel corresponding to the convolutional layer.

5. The method as described in claim 1, characterized in that, A feature processing subnetwork contains regular convolutional layers and downsampling convolutional layers; Each convolutional layer in this feature processing subnetwork sequentially processes the split image features input to the subnetwork, specifically including: The segmented image features input to the feature processing subnetwork are convolved by ordinary convolutional layers in the feature processing subnetwork, and the size and number of channels of the segmented image features input to the feature processing subnetwork are reduced by downsampling convolutional layers in the feature processing subnetwork.

6. The method as described in claim 5, characterized in that, The feature fusion subnet includes: a splicing layer, a fusion layer, and a third processing layer; The feature fusion subnetwork fuses the input split image features to obtain fused image features. Specifically, the stitching layer stitches together the split image features processed by each feature processing subnetwork to obtain stitched image features. The initial image features are processed by the third processing layer to reduce the size and number of channels of the initial image features, thereby obtaining downsampled image features. The initial image features are extracted by the extraction layer in the image feature extraction subnet, and the size and number of channels of the downsampled image features are the same as the size and number of channels of the stitched image features. The stitched image features and the downsampled image features are fused together through the fusion layer to obtain fused image features.

7. An image classification device, characterized in that, include: The acquisition module acquires the image to be processed. The feature extraction module inputs the image to be processed into the image feature extraction subnetwork in the pre-trained image classification model, so as to extract the image features to be processed from the image to be processed through the image feature extraction subnetwork; The splitting module splits the features of the image to be processed to obtain at least two split image features; The feature processing module inputs each split image feature into each feature processing subnet in the image classification model, so that the input split image features are processed in parallel through each feature processing subnet. Each feature processing subnet includes several convolutional layers. The features of each input split image are processed in parallel through a feature processing subnetwork, specifically including: For each feature processing subnetwork, the split image features input to the feature processing subnetwork are processed sequentially through each convolutional layer in the feature processing subnetwork. The sum of the number of channels of each split image feature after processing by each feature processing subnetwork is the same as the number of channels of the image feature to be processed. The fusion module inputs each split image feature processed by the feature processing subnetwork into the feature fusion subnetwork in the image classification model, so as to fuse the input split image features through the feature fusion subnetwork to obtain fused image features; The classification module classifies the image to be processed based on the fused image features.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 6.

9. An unmanned driving device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 6.

Images