Image classification method and device, computer program product, and electronic device

Abstract

The present application discloses an image classification method and apparatus, a computer program product, and an electronic device. The method comprises: acquiring a target image; analyzing the target image using at least two pre-trained classification models to obtain a first predicted classification result output by each classification model; when the first predicted classification results output by each classification model are consistent, using the first predicted classification result as the first classification result of the target image; when the first predicted classification results output by each classification model are inconsistent, performing a first processing operation on the target image, and analyzing the processed target image using at least two classification models to obtain a second predicted classification result output by each classification model, and determining the first classification result of the target image based on the first predicted classification result and the second predicted classification result output by each classification model. The present application solves the technical problem of low accuracy and high cost in classifying high-definition images due to related technologies.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to an image classification method and device, a computer program product, and an electronic device. Background Art

[0002] In common image classification tasks, images typically have hundreds of thousands of pixels, while ultra-high-definition images can have hundreds of millions of pixels, exceeding the processing capabilities of existing image classification networks. To classify these ultra-high-definition images, researchers typically use a sliding window approach to process these large pixel images.

[0003] Specifically, the sliding window method divides a large pixel image into multiple smaller windows, such as a window size of (1024, 1024). In an image with hundreds of millions of pixels, even so, an ultra-high-definition image will generate dozens or even hundreds of segmented regions, and these segmented regions are treated as independent image blocks for processing and classification. However, the above method still has the following challenges and limitations: First, the sliding window method requires a lot of computing resources and storage space to process a large number of image blocks, which leads to long training time and high computational cost; second, because the global information of the image is cut off, the classification performance will be reduced, and the accuracy of the final classification result will be poor.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present application provide an image classification method and apparatus, a computer program product, and an electronic device to at least solve the technical problems of low accuracy and high cost in classifying high-definition images due to related technologies.

[0006] According to one aspect of an embodiment of the present application, an image classification method is provided, including: obtaining a target image to be classified; analyzing the target image using at least two pre-trained classification models to obtain a first predicted classification result output by each classification model, wherein each classification model includes at least: a STEM layer, multiple MBConv modules connected in series, a pooling layer, and a fully connected layer, and the number of MBConv modules contained in each classification model is different; when the first predicted classification results output by each classification model are consistent, the first predicted classification result is used as the first classification result of the target image; when the first predicted classification results output by each classification model are inconsistent, performing a first processing operation on the target image, and analyzing the processed target image using at least two classification models to obtain a second predicted classification result output by each classification model, and determining the first classification result of the target image based on the first predicted classification result and the second predicted classification result output by each classification model, wherein the first processing operation includes at least one of the following: left-right inversion, and rotation by a preset angle.

[0007] Optionally, the training process of each classification model includes: obtaining multiple groups of training samples, wherein each group of training samples includes: image samples containing annotation information and classification results corresponding to the annotation information; constructing an initial neural network model; and iteratively training the initial neural network model using multiple groups of training samples to obtain a classification model.

[0008] Optionally, obtaining multiple groups of training samples includes: obtaining multiple original images; for each original image, segmenting the original image using preset annotation information to obtain a local image area containing the annotation information, and performing a second processing operation on the local image area to obtain corresponding image samples, wherein different annotation information corresponds to different classification results, and the second processing operation includes at least one of the following: downsampling, conversion to a standard matrix form, adjusting the image size, adjusting the image angle, and image numerical type conversion; and forming multiple groups of training samples from multiple image samples and the classification results corresponding to the annotation information in the image samples.

[0009] Optionally, the initial neural network model is iteratively trained using multiple groups of training samples to obtain a classification model, including: for each group of training samples, using the STEM layer to extract features from the image samples to obtain a first feature map; using multiple MBConv modules to perform deep feature extraction on the first feature map in sequence to obtain a second feature map; using the pooling layer to integrate and globally average pool the second feature map to obtain a first global feature vector; using the fully connected layer to map the first global feature vector to the category space to obtain the probability that the image sample belongs to each classification result in the category space, and taking the classification result with a probability not lower than a preset probability threshold as the predicted classification result of the image sample; using the classification results in each training sample and the predicted classification result to construct a target loss function, and adjusting the model parameters of the initial neural network model according to the target loss function until the model converges to obtain a classification model.

[0010] Optionally, the MBConv module includes at least: a depthwise separable convolution layer, a two-dimensional convolution layer, and an SE module, wherein multiple MBConv modules are used to sequentially perform depth feature extraction on the first feature map to obtain a second feature map, including: for the input feature map input into each MBConv module, performing the following steps: using a two-dimensional convolution layer to increase the number of feature channels of the input feature map; using a depthwise separable convolution layer to perform spatial feature extraction on the input feature map with the increased number of feature channels to obtain a third feature map; using the SE module to process the third feature map to obtain a fourth feature map; using a two-dimensional convolution layer to reduce the number of feature channels of the fourth feature map, and performing a residual connection between the fourth feature map with the reduced number of feature channels and the input feature map to obtain an input feature map of the next MBConv module; and using the input feature map of the next MBConv module output by the last MBConv module in the multiple MBConv modules as the second feature map.

[0011] Optionally, the SE module includes: a global average pooling layer and two fully connected layers, wherein the SE module is used to process the third feature map to obtain a fourth feature map, including: using the global average pooling layer to spatially compress the third feature map to obtain a second global feature vector containing only the feature channel dimension; using the first fully connected layer to compress the number of feature channels in the second global feature vector, and performing a nonlinear transformation on the compressed second global feature vector through the ReLU activation function; using the second fully connected layer to restore the number of feature channels in the second global feature vector after the nonlinear transformation, and determining the attention weights of each feature channel in the second global feature vector through the Sigmoid activation function; and obtaining the corresponding fourth feature map based on the attention weights of each feature channel and the third feature map.

[0012] Optionally, the first classification result of the target image is determined based on the first predicted classification result and the second predicted classification result output by each classification model, including: determining the occurrence frequency of each predicted classification result within the first predicted classification result and the second predicted classification result output by each classification model; and taking the predicted classification result whose occurrence frequency is not lower than a preset frequency threshold as the first classification result of the target image.

[0013] According to another aspect of an embodiment of the present application, an image classification device is also provided, including: an acquisition module for acquiring a target image to be classified; a classification result prediction module for analyzing the target image using at least two pre-trained classification models to obtain a first predicted classification result output by each classification model, wherein each classification model includes at least: a STEM layer, multiple serially connected MBConv modules, a pooling layer, and a fully connected layer, and the number of MBConv modules contained in each classification model is different; a first determination module for using the first predicted classification result as the first classification result of the target image when the first predicted classification results output by each classification model are consistent; a second determination module for performing a first processing operation on the target image when the first predicted classification results output by each classification model are inconsistent, and analyzing the processed target image using at least two classification models to obtain a second predicted classification result output by each classification model, and determining the first classification result of the target image based on the first predicted classification result and the second predicted classification result output by each classification model, wherein the first processing operation includes at least one of the following: left-right inversion and rotation by a preset angle.

[0014] According to another aspect of an embodiment of the present application, a computer program product is further provided. The computer program product includes a stored computer program, wherein the computer program implements the above-mentioned image classification method when executed by a processor.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned image classification method through the computer program.

[0016] In an embodiment of the present application, at least two classification models are used to analyze the target image to be classified, and a first predicted classification result output by each classification model is obtained; when the first predicted classification results output by each classification model are consistent, the first predicted classification result is used as the first classification result of the target image; when the first predicted classification results output by each classification model are inconsistent, a first processing operation is performed on the target image, and the processed target image is analyzed using at least two classification models to obtain a second predicted classification result output by each classification model. Based on the first predicted classification result and the second predicted classification result output by each classification model, a voting fusion technology is used to determine the first classification result of the target image. This multi-dimensional prediction method can fully classify the target image and avoid the risk of inaccurate final prediction results due to model prediction errors, thereby solving the technical problems of low accuracy and high cost in classifying high-definition images due to related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of an optional computer terminal (or mobile device) for implementing the image classification method according to an embodiment of the present application;

[0019] Figure 2 is a flowchart of an optional image classification method according to an embodiment of the present application;

[0020] Figure 3 is a schematic structural diagram of an optional MBConv module according to an embodiment of the present application;

[0021] Figure 4 is a schematic structural diagram of an optional image classification device according to an embodiment of the present application;

[0022] Figure 5 It is a schematic structural diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] In addition, the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0026] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0027] The EfficientNet model is an efficient and accurate image classification model. Its design considers the joint scaling of three dimensions: resolution (r), network depth (depth), and channel width (width). It uses NAS (Neural Architecture Search) technology to search for EfficientNet-B0 as the baseline model, and then adjusts and optimizes it based on this.

[0028] Depth-wise Separable Convolution (DSConv): A convolution operation used to reduce the computational complexity and model parameter count in convolutional neural networks. It can be split into depthwise (DW) convolution and pointwise (PW) convolution. Depthwise convolution uses a separate (k*k) convolution kernel to perform convolution calculations on each input channel (or feature map) to obtain a corresponding output channel; while pointwise convolution uses a (1*1) convolution kernel to linearly combine the output channels of depthwise convolution.

[0029] Mobile Inverted Residual Bottleneck Convolution (MBConv), also known as mobile reversible convolution, combines the residual structure with the attention mechanism. Its principle is: the input tensor is first convolved with a 1x1 convolution, and the dimension is appropriately increased or decreased while maintaining the resolution. This is followed by a depthwise separable convolution, and then enters the SE model (channel attention). Finally, a 1x1 convolution is performed to restore the input tensor dimension. Finally, the input tensor (residual) and the attention-introduced tensor are added channel by channel.

[0030] Subsampling: Also known as downsampling or image reduction, the principle is: if an image of size M*N is downsampled by a factor of s, a resolution image of size (M / s)*(N / s) can be obtained, where s needs to be a common divisor of M and N. If the image is in matrix form, the image within the s*s window of the original image is converted into a pixel, and the value of this pixel is the mean or maximum value of all pixels in the s*s window.

[0031] The STEM (Spatial Transformer Embedding Module) layer is a convolutional neural network module that extracts richer features by spatially transforming the feature map and fusing local features. The principle is: first, the input image is repositioned, scaled, and rotated using the adaptively deformable spatial transformer, so that the network is robust to different transformations of the input image and can automatically align features of different angles, sizes, and positions; then, the spatially transformed feature map is fused with the original image using branch connections to retain the low-level features in the original image, while the transformed feature map is used to enhance the ability to distinguish difficult samples; finally, layer-structured convolution is used to perform convolution operations on different levels of the feature map to extract richer feature representations.

[0032] Example 1

[0033] According to an embodiment of the present application, a method embodiment of an image classification method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image classification method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0035] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image classification method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the image classification method of the above-mentioned application. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0038] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0039] In the above operating environment, Figure 2 is a flow chart of an optional image classification method according to an embodiment of the present application, such as Figure 2 As shown, the method includes at least steps S202-S208, wherein:

[0040] Step S202: Acquire a target image to be classified.

[0041] In the technical solution provided in step S202, the target image may be an ordinary image or an (ultra) high-definition image, such as a lesion image, a landscape image, a person image, etc.

[0042] Step S204: Analyze the target image using at least two pre-trained classification models to obtain a first predicted classification result output by each classification model.

[0043] In the technical solution provided in step S204, the model architecture of each of the above classification models includes at least: a STEM layer, multiple serially connected MBConv modules, a pooling layer, and a fully connected layer, and the difference between the various classification models includes at least: a different number of reused MBConv modules.

[0044] Step S206 : When the first predicted classification results output by each classification model are consistent, the first predicted classification result is used as the first classification result of the target image.

[0045] Step S208: When the first predicted classification results output by each classification model are inconsistent, a first processing operation is performed on the target image, and the processed target image is analyzed using at least two classification models to obtain the second predicted classification results output by each classification model, and the first classification result of the target image is determined based on the first predicted classification results and the second predicted classification results output by each classification model.

[0046] In the technical solution provided in step S208, in order to select the most realistic classification result from the predicted classification results output by each classification model, the present application can perform a first processing operation on the target image, which includes but is not limited to: left-right inversion, rotation by a preset angle, etc., to change certain attributes of the image, making the difficult-to-classify target image easier to recognize; then, the processed target image is analyzed using at least two classification models to obtain the second predicted classification results output by each classification model; finally, the first classification result of the target image is determined from the first predicted classification results and the second predicted classification results output by each classification model. It should be noted that the embodiment of the present application does not specifically limit the execution order of steps S206 and S208.

[0047] The above method of this embodiment is further introduced below.

[0048] As an optional implementation, in the technical solution provided in step S204 above, the training process of each classification model includes:

[0049] Step S1, obtaining multiple sets of training samples, wherein each set of training samples includes: image samples containing annotation information and classification results corresponding to the annotation information;

[0050] Step S2, constructing an initial neural network model;

[0051] Step S3, iteratively training the initial neural network model using multiple sets of training samples to obtain a classification model.

[0052] Optionally, the technical solution provided in step S1 above can be implemented by the following steps:

[0053] Step S11, obtaining multiple original images.

[0054] The original image and the first image are of the same image type, that is, if the first image is a lesion image, the original image is also a lesion image; if the first image is a person image, the original image is also a person image.

[0055] Step S12: for each original image, segment the original image using the preset annotation information to obtain a local image region containing the annotation information, and perform a second processing operation on the local image region to obtain a corresponding image sample.

[0056] Specifically, the original image is segmented according to the preset annotation information to obtain local image areas containing annotation information corresponding to each original image, wherein different annotation information corresponds to different classification results. For example, for lesion images, Json annotation information of different lesion levels can be used to segment the lesion image to obtain polygons containing different lesion areas; and in order to reduce the storage scale of the original image, each original image can be downsampled first, wherein the downsampling multiple can be set according to the actual application scenario, and this application does not impose specific restrictions on this; in addition, considering that the local image area obtained by segmentation may be an irregular polygon, the local image area can be filled with a value of 0 to convert it into a standardized rectangular area. At the same time, in order to ensure that original data of different formats can be converted into image samples of the same size and format, the embodiment of the present application can also perform image size adjustment, image angle adjustment, image numerical type conversion and other processing operations on the original image.

[0057] Different annotation information corresponds to different classification results, and the second processing operation includes at least one of the following: downsampling, conversion to a standard matrix form, image size adjustment, image angle adjustment, and image numerical type conversion.

[0058] Step S13: forming a plurality of training samples from the plurality of image samples and the classification results corresponding to the annotated information in the image samples.

[0059] Optionally, in the technical solution provided in step S2 above, the initial neural network model can be an EfficientNet series model, such as EfficientNet B0, B1, ..., B7, etc., and its model architecture includes the following parts: STEM layer, reused MBCconv module (including depth-separable convolution, two-dimensional convolution layer), pooling layer (including two-dimensional convolution layer and pooling layer), and fully connected layer.

[0060] Optionally, the technical solution provided in step S3 above can be implemented by the following steps:

[0061] Step S31: For each set of training samples, input the image samples in the training samples into the initial neural network model, and obtain the predicted classification results of the image samples output by the neural network model through the following steps, including:

[0062] Step S311: extract features from the image sample using the STEM layer to obtain a first feature map.

[0063] Among them, the above-mentioned STEM layer better extracts the global and contextual information of the input image sample through spatial transformation and feature fusion, as well as hierarchical convolution operations, to obtain the corresponding first feature map, wherein the convolution kernel size of the two-dimensional convolution layer in the STEM layer can be 3*3 or 5*5, and this application does not impose any specific restrictions on the selection of its convolution kernel size.

[0064] Step S312: Use multiple MBConv modules to sequentially perform deep feature extraction on the first feature map to obtain a second feature map.

[0065] The core of the initial neural network model is the stacking of multiple MBConv modules, and the structure of each MBConv module is as follows: Figure 3 As shown in the figure, it includes: a 2D convolutional layer for dimension increase, a depthwise separable convolutional layer, an SE module, and a 2D convolutional layer for dimension reduction. By repeatedly stacking multiple MBConv modules, rich feature representations can be extracted from the first feature map output by the STEM layer.

[0066] Step S313: using a pooling layer to integrate and perform global average pooling on the second feature map to obtain a global feature vector;

[0067] Specifically, a two-dimensional convolutional layer (usually a convolution kernel of size 1*1, also called Con2D) is used to integrate the second feature map to integrate it into a format suitable for global average pooling. Then, global average pooling is performed on the two-dimensional feature map to convert it into a one-dimensional first global feature vector in preparation for the final classification task.

[0068] In step S314, the first global feature vector is mapped to the category space using a fully connected layer to obtain the probability that the image sample belongs to each classification result in the category space, and the classification result with a probability not lower than a preset probability threshold is used as the predicted classification result of the image sample.

[0069] Specifically, the fully connected layer performs a linear transformation on the first global eigenvector of the input according to the following formula: z = Wx + b, where x represents the first global eigenvector of the input, W represents the weight matrix of the fully connected layer, b represents the bias vector of the fully connected layer, which adds a constant offset to the score of each category to adjust the baseline value of each category score, and z represents the vector after the linear transformation. Furthermore, the vector after the linear transformation can be nonlinearly mapped through an activation function. The activation function commonly used is the Softmax function, which can convert the linear score into a probability distribution, so that the scores of the K categories in the category space are converted into probability values ​​between 0 and 1, and the sum of the probabilities of all categories is 1. In other words, after applying the Softmax function, the score z of each category isi Converted into probabilities Its expression can be written as:

[0070]

[0071] Finally, the classification result with the highest probability or a probability not lower than a preset probability threshold is used as the predicted classification result of the image sample, and the setting of the above probability threshold can be set according to the actual application scenario.

[0072] In step S32, a target loss function is constructed using the classification results and the predicted classification results in each training sample, and the model parameters of the initial neural network model are adjusted according to the target loss function until the model converges to obtain a classification model.

[0073] The target loss function may be a cross entropy loss function or other types of loss functions, which are not specifically limited in the present embodiment. During the training process, the model parameters of the initial neural network model are optimized by minimizing the value of the loss function until a preset number of iterations is reached or the model performance meets a preset condition, so that the trained classification model can more accurately predict the image category.

[0074] Furthermore, in the technical solution provided in step S312 above, the multiple MBConv modules within the initial neural network model are connected in series. Therefore, for the first MBConv module, the feature map input thereto is the first feature map output by the MBConv module; and for MBConv modules other than the first MBConv module, the feature map input thereto is the feature map output by the previous MBConv module. Below, the feature maps input to each MBConv module are collectively referred to as "input feature maps," and the specific processing flow of the "input feature maps" by the MBConv modules is briefly described.

[0075] Optionally, for each MBConv module, its processing flow for the input feature map includes:

[0076] Step 1: Use a two-dimensional convolutional layer to increase the number of feature channels of the input feature map.

[0077] The two-dimensional convolution layer used above can be a convolution kernel with a size of 1*1. In addition, in the process of increasing the number of feature channels of the input feature map, it is usually used in combination with batch normalization and an activation function (such as ReLU or Swish).

[0078] Step 2: Use the depthwise separable convolutional layer to extract spatial features from the input feature map with increased number of feature channels to obtain the third feature map.

[0079] Among them, the depth-wise separable convolution layer includes depth-wise convolution and point-by-point convolution. The depth-wise convolution uses a convolution kernel to convolve each feature channel of the input feature map, while the point-by-point convolution uses a convolution kernel of size 1*1 to merge the results of the depth-wise convolution output.

[0080] Step 3: Use the SE module to process the third feature map to obtain the fourth feature map.

[0081] Among them, the above SE module is used to enhance the feature extraction capability of the network and redistribute the importance weights of channels by explicitly modeling the correlation between channels.

[0082] Specifically, the SE module includes: a global average pooling layer (for realizing compression function) and two fully connected layers (for realizing excitation function), and the fourth feature map can be obtained according to the following steps, including:

[0083] First, the third feature map is spatially compressed using the Global Average Pooling (GAP) layer to obtain a second global feature vector containing only the feature channel dimension, that is, a 1*1*C(Channel) channel descriptor. The channel descriptor captures the global distribution of each channel in the entire spatial range, so that the global receptive field from the network can be utilized by its lower layers.

[0084] Next, the first fully connected layer is used to compress the number of feature channels in the second global feature vector, and the compressed second global feature vector is nonlinearly transformed using the ReLU activation function. In other words, the first fully connected layer is used to compress the number of feature channels in the second global feature vector from C to C / r, where r is the compression ratio, and the compressed second global feature vector is nonlinearly transformed using the ReLU activation function.

[0085] Then, the second fully connected layer is used to restore the number of feature channels in the second global feature vector after nonlinear transformation, and the attention weight of each feature channel in the second global feature vector is determined by the Sigmoid activation function. In other words, the second fully connected layer is used to restore the number of feature channels in the second global feature vector from C / r to C, and the Sigmoid activation function is used to obtain a 1*1*C weight tensor to determine the attention weight of each feature channel in the second global feature vector.

[0086] Finally, the corresponding fourth feature map is obtained based on the attention weights of each feature channel and the third feature map. In other words, the weight tensors between the channels learned by the two fully connected layers are multiplied channel by channel by the original feature map (i.e., the third feature map) to achieve re-weighting of the feature map, thereby obtaining the fourth feature map output by the SE module. In this way, the network can learn to use global information to selectively emphasize useful features and suppress less useful features.

[0087] Step 4: Use a two-dimensional convolutional layer to reduce the number of feature channels of the fourth feature map, and perform a residual connection between the fourth feature map with reduced feature channels and the input feature map to obtain the input feature map of the next MBConv module.

[0088] Among them, the two-dimensional convolution layer used above is still a convolution kernel of size 1*1. Similarly, in the process of reducing the number of feature channels of the fourth feature map, it can also be used in combination with batch normalization and activation functions (such as ReLU or Swish). In addition, if the input feature map of the MBConv module and the input feature map have the same dimension, the input feature map can be directly added to the input feature map of the next MBConv module output by the MBConv module using a residual connection, thereby avoiding the problem of gradient disappearance.

[0089] After obtaining the input feature maps of each MBConv module through the solutions provided in the first to fourth steps above, considering the series relationship between the MBConv modules, after the first feature map is processed in sequence by multiple MBConv modules, the output second feature map should be the input feature map of the next MBConv module output by the last MBConv module in the multiple MBConv modules. Therefore, this application uses the input feature map of the next MBConv module output by the last MBConv module in the multiple MBConv modules as the second feature map.

[0090] Optionally, in step S3, the classification model can be loaded into the memory, for example, the raw data of the classification model can be loaded from the non-volatile memory into the volatile memory so that the processor runs the first neural network. The raw data of the classification model refers to unprocessed data, which usually includes the parameters and structural data of the classification model. The structural data can be a calculation relationship based on the parameters, such as the forward propagation calculation relationship between the intermediate layers and between neurons. Specifically, the structural data can include code related to the structure of the classification model, such as code for performing related calculations between the intermediate layers and between neurons. In one embodiment, an area for loading the classification model can be divided in the memory, which can include a structural data storage area and a parameter storage area. The structural data storage area is used to store structure-related code, and the parameters referenced therein can point to the address of specific parameters in the parameter storage area through a pointer. During the training process of the classification model, it may be necessary to update the parameters frequently, and the parameter values ​​in the parameter storage area can be updated.

[0091] The following will take the target image as a lesion image as an example to briefly describe the classification model construction process of the embodiment of the present application.

[0092] Step 1: Obtain a sample set as follows:

[0093] Multiple ultra-high-definition lesion images with annotated information are obtained from a preset database. The format can be kfb format, and the annotated information includes the lesion area and category. The database can be divided into five lesion categories as listed in Table 1 below, denoted as: T0, T1, T2, T3, and Tis, where: T0 indicates no lesion symptoms and no annotated area, while T1-Tis respectively indicate categories of progressively more severe cancerous changes, and therefore have annotated lesion areas and category information. In addition, since these five categories of images are all ultra-high-definition images, the storage space occupied by the database when storing the above five categories of image data is very large.

[0094] To reduce image space usage, the following steps can be performed for each ultra-high-definition lesion image: First, the image is segmented based on the annotation information to obtain irregular polygons containing the lesion area; then, the irregular polygons containing the lesion area are downsampled by a factor of 16 and filled with standard rectangular areas. These steps ensure that the shape distribution of each image is consistent with that of the original image, and the storage space occupied by the database when storing the segmented image data is greatly reduced.

[0095] Furthermore, in order to increase the robustness of the final trained classification model, the present application can also preprocess the segmented images, including: image resizing (Resize), random up and down flipping (RandomVerticalFilp), random affine transformation (RandomAffine), data type tensor (Totensor), normalization (Normalize), etc., to obtain multiple groups of training samples containing image samples and sample labels (i.e., the categories corresponding to the annotation information).

[0096] Step 2: Use the Efficient Net-B0 and Efficient Net-B4 models as the initial neural network models.

[0097] Step 3: Use the sample set to iteratively train the Efficient Net-B0 and Efficient Net-B4 models to obtain two classification models.

[0098] When training Efficient Net-B0, the Adam optimizer can be used with an initial learning rate of 0.001, trained for 30 batches, and a learning rate adjustment rule of linear descent, reducing the learning rate by 0.1 times every 10 batches. When training EfficientNet-B4, the Adam-W optimizer can be used with an initial learning rate of 0.001, trained for 80 batches, and a cosine annealing learning rate adjustment strategy with a minimum value of 0.001 times the initial value.

[0099] By processing the target image using the classification model trained by the above steps S1-S3, the first predicted classification results output by each classification model can be accurately obtained. If the first predicted classification results output by each classification model are inconsistent, the embodiment of the present application proposes that the target image can be expanded (such as rotated, flipped, etc.) to change certain attributes of the image; then, the processed target image is analyzed again using the classification model, so that each classification model can capture different features of the target image, so as to obtain the second predicted classification results output by each classification model based on the extracted image feature information.

[0100] As an optional implementation, in the technical solution provided in step S208 above, the first classification result of the target image can be determined from the first predicted classification results and the second predicted classification results output by each classification model by the following method, including:

[0101] Step 1: Determine the occurrence frequency of each predicted classification result in the first predicted classification result and the second predicted classification result output by each classification model;

[0102] Step 2: The predicted classification results whose occurrence frequency is not less than the preset frequency threshold are used as the first classification results of the target image.

[0103] That is to say, by counting the occurrence frequencies of different predicted classification results in the first predicted classification results and the second predicted classification results output by each classification model, the predicted classification result with the highest occurrence frequency is used as the first classification result of the target image, so that the final first classification result is more accurate and robust.

[0104] Based on the scheme defined by the above steps S202 to S208, it can be known that, in an embodiment, by using at least two classification models to analyze the target image to be classified, a first predicted classification result output by each classification model is obtained; when the first predicted classification results output by each classification model are consistent, the first predicted classification result is used as the first classification result of the target image; when the first predicted classification results output by each classification model are inconsistent, a first processing operation is performed on the target image, and the processed target image is analyzed using at least two classification models to obtain a second predicted classification result output by each classification model. Based on the first predicted classification result and the second predicted classification result output by each classification model, a voting fusion technology is used to determine the first classification result of the target image. This multi-dimensional prediction method can fully classify the target image and avoid the risk of inaccurate final prediction results due to model prediction errors, thereby solving the technical problems of low accuracy and high cost in classifying high-definition images due to related technologies.

[0105] Example 2

[0106] Based on the embodiment 1 of the present application, an embodiment of an image classification device is also provided, which executes the above-mentioned image classification method of the above-mentioned embodiment when the device is running. Figure 4 is a schematic structural diagram of an optional image classification device according to an embodiment of the present application, such as Figure 4 As shown, the image classification device at least includes an acquisition module 42, a classification result prediction module 44, a first determination module 46 and a second determination module 48, wherein:

[0107] An acquisition module 42 is used to acquire a target image to be classified;

[0108] a classification result prediction module 44 for analyzing the target image using at least two pre-trained classification models to obtain a first predicted classification result output by each classification model, wherein each classification model includes at least: a STEM layer, multiple serially connected MBConv modules, a pooling layer, and a fully connected layer, and the number of MBConv modules included in each classification model varies;

[0109] A first determination module 46 is configured to use the first predicted classification result as the first classification result of the target image when the first predicted classification results output by the classification models are consistent;

[0110] The second determination module 48 is used to perform a first processing operation on the target image when the first predicted classification results output by each classification model are inconsistent, and use at least two classification models to analyze the processed target image to obtain the second predicted classification results output by each classification model, and determine the first classification result of the target image based on the first predicted classification results and the second predicted classification results output by each classification model, wherein the first processing operation includes at least one of the following: left-right inversion, rotation by a preset angle.

[0111] It should be noted that the various modules in the above-mentioned image classification device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0112] Example 3

[0113] According to an embodiment of the present application, a computer program product is further provided. The computer program product includes a stored computer program, wherein when the computer program is executed by a processor, the image classification method in Example 1 is implemented.

[0114] Optionally, the computer program executes the following steps: obtaining a target image to be classified; analyzing the target image using at least two pre-trained classification models to obtain a first predicted classification result output by each classification model, wherein each classification model includes at least: a STEM layer, multiple MBConv modules connected in series, a pooling layer, and a fully connected layer, and the number of MBConv modules contained in each classification model is different; when the first predicted classification results output by each classification model are consistent, the first predicted classification result is used as the first classification result of the target image; when the first predicted classification results output by each classification model are inconsistent, performing a first processing operation on the target image, and analyzing the processed target image using at least two classification models to obtain a second predicted classification result output by each classification model, and determining the first classification result of the target image based on the first predicted classification result and the second predicted classification result output by each classification model, wherein the first processing operation includes at least one of the following: left-right inversion, rotation by a preset angle.

[0115] According to an embodiment of the present application, an electronic device is further provided, wherein: Figure 5 is a schematic structural diagram of an optional electronic device according to an embodiment of the present application, such as Figure 5The electronic device includes one or more processors; a memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the image classification method in the above-mentioned embodiment 1 when running.

[0116] Optionally, the processor is configured to execute the following steps through a computer program: obtaining a target image to be classified; analyzing the target image using at least two pre-trained classification models to obtain a first predicted classification result output by each classification model, wherein each classification model includes at least: a STEM layer, multiple serially connected MBConv modules, a pooling layer, and a fully connected layer, and the number of MBConv modules contained in each classification model is different; when the first predicted classification results output by each classification model are consistent, the first predicted classification result is used as the first classification result of the target image; when the first predicted classification results output by each classification model are inconsistent, performing a first processing operation on the target image, and analyzing the processed target image using at least two classification models to obtain a second predicted classification result output by each classification model, and determining the first classification result of the target image based on the first predicted classification result and the second predicted classification result output by each classification model, wherein the first processing operation includes at least one of the following: left-right inversion, rotation by a preset angle.

[0117] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0118] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0120] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0121] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0122] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0123] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image classification method, characterized in that: include: Obtain the target image to be classified; The target image is analyzed using at least two pre-trained classification models to obtain a first predicted classification result output by each of the classification models, wherein each of the classification models includes at least: a spatial transformer embedding module STEM layer, multiple serially connected moving flip bottleneck convolution MBConv modules, a pooling layer, and a fully connected layer, and the number of MBConv modules included in each of the classification models is different. During the training process of the classification model, the STEM layer is used to extract features from the input sample image through spatial transformation, feature fusion, and hierarchical convolution to obtain a first feature map, and the MBConv module is used to sequentially perform deep feature extraction on the first feature map; When the first predicted classification results output by the classification models are consistent, using the first predicted classification results as the first classification result of the target image; When the first predicted classification results output by each of the classification models are inconsistent, a first processing operation is performed on the target image, and the target image processed by the first processing operation is analyzed using at least two of the classification models to obtain a second predicted classification result output by each of the classification models, and the first classification result of the target image is determined based on the first predicted classification result and the second predicted classification result output by each of the classification models, wherein the first processing operation includes at least one of the following: left-right inversion, and rotation by a preset angle.

2. The method according to claim 1, characterized in that The training process of each classification model includes: Acquire multiple groups of training samples, wherein each group of training samples includes: image samples containing annotation information and classification results corresponding to the annotation information; Build an initial neural network model; The initial neural network model is iteratively trained using multiple groups of training samples to obtain the classification model.

3. The method according to claim 2, characterized in that Get multiple sets of training samples, including: Acquire multiple original images; For each of the original images, segment the original image using preset annotation information to obtain a local image region containing the annotation information, and perform a second processing operation on the local image region to obtain the corresponding image sample, wherein different annotation information corresponds to different classification results, and the second processing operation includes at least one of the following: downsampling, conversion to a standard matrix form, image resizing, image angle adjustment, and image numerical type conversion; A plurality of groups of training samples are formed by a plurality of the image samples and classification results corresponding to the labeled information in the image samples.

4. The method according to claim 3, characterized in that Iteratively training the initial neural network model using multiple sets of training samples to obtain the classification model includes: For each group of training samples, the STEM layer is used to perform feature extraction on the image samples to obtain a first feature map; multiple MBConv modules are used to sequentially perform deep feature extraction on the first feature map to obtain a second feature map; the pooling layer is used to integrate and perform global average pooling on the second feature map to obtain a first global feature vector; the fully connected layer is used to map the first global feature vector to a category space to obtain the probability that the image sample belongs to each classification result in the category space, and the classification result with the probability not less than a preset probability threshold is used as the predicted classification result of the image sample; The target loss function is constructed using the classification results and the predicted classification results in each of the training samples, and the model parameters of the initial neural network model are adjusted according to the target loss function until the model converges to obtain the classification model.

5. The method according to claim 4, characterized in that The MBConv module includes at least: a depthwise separable convolution layer, a two-dimensional convolution layer, and a compression and excitation network SE module, wherein the first feature map is sequentially subjected to deep feature extraction using multiple MBConv modules to obtain a second feature map, including: For the input feature map input into each MBConv module, the following steps are performed: using the two-dimensional convolution layer to increase the number of feature channels of the input feature map; using the depthwise separable convolution layer to perform spatial feature extraction on the input feature map with the increased number of feature channels to obtain a third feature map; using the SE module to process the third feature map to obtain a fourth feature map; using the two-dimensional convolution layer to reduce the number of feature channels of the fourth feature map, and performing a residual connection between the fourth feature map with the reduced number of feature channels and the input feature map to obtain an input feature map of the next MBConv module; The input feature map of the next MBConv module output by the last MBConv module among the multiple MBConv modules is used as the second feature map.

6. The method according to claim 5, characterized in that The SE module includes: a global average pooling layer and two fully connected layers, wherein the SE module is used to process the third feature map to obtain a fourth feature map, including: Using the global average pooling layer to spatially compress the third feature map to obtain a second global feature vector containing only feature channel dimensions; Using the first fully connected layer to compress the number of feature channels in the second global feature vector, and performing a nonlinear transformation on the compressed second global feature vector through a ReLU activation function; Restoring the number of feature channels in the second global feature vector after the nonlinear transformation using the second fully connected layer, and determining the attention weight of each feature channel in the second global feature vector using a Sigmoid activation function; According to the attention weights of the feature channels and the third feature map, the corresponding fourth feature map is obtained.

7. The method according to claim 1, characterized in that Determining a first classification result of the target image based on the first predicted classification results and the second predicted classification results output by each of the classification models includes: Determining the occurrence frequency of each predicted classification result in the first predicted classification result and the second predicted classification result output by each of the classification models; The predicted classification result whose occurrence frequency is not lower than a preset frequency threshold is used as the first classification result of the target image.

8. An image classification device, characterized in that: include: An acquisition module, used to acquire a target image to be classified; a classification result prediction module, configured to analyze the target image using at least two pre-trained classification models to obtain a first predicted classification result output by each of the classification models, wherein each of the classification models includes at least: a STEM layer, multiple serially connected MBConv modules, a pooling layer, and a fully connected layer, and each of the classification models includes a different number of MBConv modules; during the training process of the classification model, the STEM layer is configured to extract features from the input sample image through spatial transformation, feature fusion, and hierarchical convolution to obtain a first feature map, and the MBConv module is configured to sequentially perform deep feature extraction on the first feature map; a first determining module, configured to use the first predicted classification results as the first classification result of the target image when the first predicted classification results output by the classification models are consistent; The second determination module is used to perform a first processing operation on the target image when the first predicted classification results output by each of the classification models are inconsistent, and use at least two of the classification models to analyze the target image after the first processing operation to obtain the second predicted classification results output by each of the classification models, and determine the first classification result of the target image based on the first predicted classification results and the second predicted classification results output by each of the classification models, wherein the first processing operation includes at least one of the following: left-right inversion, and rotation by a preset angle.

9. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the image classification method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the image classification method according to any one of claims 1 to 7 is executed when the program is run.

Images