INTELLIGENT DIAGNOSTIC SUPPORT METHOD AND TERMINAL DEVICE BASED ON MEDICAL IMAGES

Description

TECHNICAL FIELD

[0001] The present invention involves in the field of computer technologies, and particularly relates to a smart diagnosis assistance method based on medical images and a smart diagnosis assistance terminal based on medical images.BACKGROUND

[0002] At present, with the development of society, performing smart assistance diagnosis for medical image classification based on a deep network model is applied more and more extensively. However, when a medical image is processed by a traditional deep network model, internal information of tissue structures and correlation information between different regions are lost, and the loss of the spatial structure information will eventually lead to an inaccurate classification result, thereby seriously affecting performance of a smart assistance diagnosis model for diseases. For example, a prior art document US2019197368A1 discloses Mechanisms to implement a generative adversarial network (GAN) that is trained based on labeled image data, unlabeled image data, and generated image data generated by a generator of the GAN. The GAN comprises a loss function that comprises error components for each of the labeled image data, unlabeled image data, and generated image data which is used to train the GAN. A new data source for which the trained GAN is to be adapted is identified and the trained GAN is adapted for the new data source. Image data in the new data source is classified by applying the adapted GAN to the data in the new data source. Adapting the trained GAN includes obtaining a minimized set of labeled images and utilizing the minimized set of images to perform the adapting of the trained GAN. A prior art document ZILIN GAO ET AL: "Global Second-order Pooling Convolutional Networks", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 29 November 2018, discloses a global second-order Pooling Convolutional networks, which introduces GSoP across lower to higher layers for exploiting holistic image information throughout a network, the GSoP is performed for tensor scaling along channel dimension and spatial dimension. A prior art document Li Chongxuan ET AL: "Triple Generative Adversarial Nets", arxiv.org, 5 November 2017 (2017-11-05), pages 1-15 discloses a Triple Generative Adversarial Nets (Triple-GAN), which consists of three players-a generator, a discriminator and a classifier. The generator and the classifier characterize the conditional distributions between images and labels, and the discriminator solely focuses on identifying fake image-label pairs. A prior art document CN108198179A discloses a generative adversarial network improved CT medical image pulmonary nodule detection method. The method includes: acquiring a section of a pulmonary CT image; separating according to image morphological properties to acquire a ROI pulmonary parenchyma area; acquiring different suspected pulmonary nodule candidate sets according to a connected domain formed by a binarized image; building a model of an assistant classifier generative adversarial network to generate positive samples overcome the circumstance that positive-negative sample number is unbalanced; building a convolution neural network to classify suspected pulmonary nodule parts to acquire pulmonary nodule areas; using a non-maximum suppression algorithm to acquire a final area of pulmonary nodule. According to this method, efficient processing performance of a computer can be fully utilized, and data processing efficiency is improved; through a convolution neural network algorithm, classification accuracy is improved, and processing performance of CT image data is improved.TECHNICAL PROBLEM

[0003] One of purposes of embodiments of the present invention is to provide a smart diagnosis assistance method based on medical images and a smart diagnosis assistance terminal based on medical images, so as to solve the problem of the inaccurate classification result due to that a lot of internal structure information and internal correlation information of the image are lost when a medical image is processed by a traditional deep network model.TECHNICAL SOLUTIONS

[0004] In order to solve the above technical problem, technical solutions adopted by the present invention are as follows.

[0005] A first aspect provides a smart diagnosis assistance method based on medical images, which includes: acquiring a medical image to be classified; preprocessing the medical image to be classified to obtain a preprocessed image; inputting the preprocessed image into a trained classification model for classification processing to obtain a classification category corresponding to the preprocessed image; here the trained classification model is a triple generative adversarial nets obtained by training sample images and classification categories corresponding to the sample images based on a preset generator model, a preset discriminator model and a preset classifier model; the trained classification model includes a trained classifier model, the trained classifier model includes a tensorized network layer and a second-order pooling module; the inputting the preprocessed image into the trained classification model for classification processing to obtain the classification category corresponding to the preprocessed image includes: normalizing the preprocessed image by using the classifier model to obtain a target image; extracting key features in the target image based on the tensorized network layer and the second-order pooling module of the trained classifier model to obtain a global high-order feature map; acquiring the classification category corresponding to the global high-order feature map by using the trained classifier model; the extracting the key features in the target image based on the tensorized network layer and the second-order pooling module of the trained classifier model to obtain the global high-order feature map includes: extracting features in the target image through the tensorized network layer in the trained classifier model to obtain a first feature map; performing channel dimension reduction on the first feature map through the second-order pooling module in the trained classifier model to obtain a dimension-reduced second feature map; calculating a weight vector corresponding to the second feature map; and weighting the first feature map based on the weight vector to obtain the global high-order feature map; before acquiring the medical image to be classified, the method further includes: training the sample images and the classification categories corresponding to the sample images based on the preset generator model, the preset discriminator model and the preset classifier model to obtain the triple generative adversarial nets; wherein each network layer in the preset generator model, the preset discriminator model and the preset classifier model is tensorized during a training process of the classification module; and acquiring the trained classifier model from the triple generative adversarial nets.

[0006] A second aspect provides a smart diagnosis assistance terminal based on medical images, characterized in that, comprising: an acquisition unit configured to acquire a medical image to be classified; a preprocessing unit configured to preprocess the medical image to be classified to obtain a preprocessed image; a classification unit configured to input the preprocessed image into a trained classification model for classification processing to obtain a classification category corresponding to the preprocessed image; here the classification model is a triple generative adversarial nets obtained by training sample images and classification categories corresponding to the sample images based on a preset generator model, a preset discriminator model and a preset classifier model; the trained classification model includes a trained classifier model, the trained classifier model includes a tensorized network layer and a second-order pooling module, and the classification unit includes: a processing unit configured to normalize the preprocessed image by using the classifier model to obtain a target image; an extraction unit configured to extract key features in the target image based on the tensorized network layer and the second-order pooling module of the trained classifier model to obtain a global high-order feature map; a classification category acquisition unit configured to acquire the classification category; the extraction unit is specifically configured to: extract features in the target image through the tensorized network layer in the classifier model to obtain a first feature map; perform channel dimension reduction on the first feature map through the second-order pooling module in the classifier model to obtain a dimension-reduced second feature map; calculate a weight vector corresponding to the second feature map; weight the first feature map based on the weight vector to obtain the global high-order feature map; the terminal further includes: a training unit configured to train the sample images and the classification categories corresponding to the sample images based on the preset generator model, the preset discriminator model and the preset classifier model to obtain the triple generative adversarial nets; wherein each network layer in the preset generator model, the preset discriminator model and the preset classifier model is tensorized during a training process of the classification module; and a model acquisition unit configured to acquire the trained classifier model from the triple generative adversarial nets.

[0007] A third aspect provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements following steps of: acquiring a medical image to be classified; preprocessing the medical image to be classified to obtain a preprocessed image; inputting the preprocessed image into a trained classification model for classification processing to obtain a classification category corresponding to the preprocessed image; here the classification model comprises a tensorized network layer and a second-order pooling module, the classification model is a triple generative adversarial nets obtained by training sample images and classification categories corresponding to the sample images based on a preset generator model, a preset discriminator model and a preset classifier model. BENEFICIAL EFFECT

[0008] In the embodiments of the present invention, the medical image to be classified is obtained through the terminal; the medical image to be classified is preprocessed to obtain the preprocessed image; and the preprocessed image is classified based on the trained classification model to acquire the corresponding classification result. Since the trained classification model includes the tensor-decomposited network layer and the second-order pooling module, the internal structure information and the internal correlation of the medical image are preserved when the medical image is processed based on this classification model, and the second-order pooling module, based on the dependencies of different regions of the medical image and the correlation information between different channels of high-order features, makes the weight of the important feature channel larger and the weight of the unimportant feature channel smaller under the action of the self-attention mechanism, so as to extract better discriminative features associated with a lesion, thereby improving the accuracy of the smart assistance diagnosis for diseases. Further, in the present invention, the images are classified based on the triple generative adversarial nets, which, on the basis of the traditional binary generative adversarial network including a generator and a discriminator, adds up the classifier network model, and alleviates the problem of training instability of the binary generative adversarial network through designing a compatible loss function, solves the problem of inconsistent convergence points of the discriminator in the traditional binary generative adversarial network when performing two target tasks of distinguishing true and false and classifying, so that the generator model and the classifier model work in a cooperative training manner to replace the generative adversarial training method of the traditional binary generative adversarial network, thereby enabling the generator model and the classifier model to achieve the best optimality at the same time, speeding up the convergence, and making the generative adversarial network easier to reach the Nash equilibrium. In the present invention, when the medical images are classified, the tensorization method instead of the traditional vectorization method is used to compress each network layer in the triple generative adversarial nets, which achieves the regularization effect on the network model while reducing the parameters, and solves the problems of excessive parameters and overfitting in classification and recognition of high-resolution images; further, the tensorization method can maintain internal correlation between the internal spatial structure information and different voxels, and solves the problem of information loss of the internal structure of the vectorized network layers. In the present invention, the traditional first-order pooling (maximum pooling or average pooling) is replaced by the second-order pooling, and the better discriminative features are automatically extracted under the action of the self-attention mechanism by making full use of the second-order information of the overall image, thereby improving the classification accuracy of the classifier. The triple generative adversarial nets in the present invention uses the semi-supervised learning method to jointly cooperate and train the preset generator model, the preset discriminator model and the preset classifier model, this reduces the network model's demand for image labeling information and makes full use of unlabeled data, thereby realizing the intelligent network model provided with high precision and high robustness; moreover, this reduces the number of the parameters, improves the computing efficiency, helps reduce the performance requirements of the terminal, thereby accelerating the disease diagnosis process for patients.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the drawings needed to be used in description for the embodiments or exemplary technologies are briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and other drawings may also be obtained for those of ordinary skill in the art based on these drawings without any creative effort. FIG. 1 is a flowchart of an implementation of a smart diagnosis assistance method based on medical images provided by an embodiment of the present invention. FIG. 2 is a structural schematic diagram of a trained classifier model provided by the present invention. FIG. 3 is a structural schematic diagram of a second-order pooling module provided by the present invention. FIG. 4 is a flowchart of an implementation of a smart diagnosis assistance method based on medical images provided by another embodiment of the present invention. FIG. 5 is a structural schematic diagram of a triple generative adversarial nets provided by the present invention. FIG. 6 is a schematic diagram of a smart diagnosis assistance terminal based on medical images provided by an embodiment of the present invention. IMPLEMENTATION OF THE INVENTION

[0010] In order to make the purposes, technical solutions and advantages of the present invention clearer and more comprehensible, the present invention will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

[0011] It should be noted that, when a component is referred to as being "fixed to" or "disposed on" another component, this component may be directly on the another component or indirectly on the another component; when a component is referred to as being "connected to" another component, this component may be directly or indirectly connected to the another component. Orientations or positional relationships indicated by the terms such as "upper", "lower", "left", "right" etc. are based on orientations or positional relationships as shown in the drawings, which are only intended to facilitate describing rather than to indicate or imply that a specified device or component must have a specific orientation or be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present invention. The above terms may be understood according to specific situations for those of ordinary skill in the art. The terms "first" and "second" are only used for a descriptive purpose, and cannot be understood as an indication or implication of relative importance or an implicit indication of the number of a specified technical feature. The phrase "a plurality of" means two or more, unless otherwise explicitly and specifically defined.

[0012] In order to illustrate the technical solutions of the present invention, a detailed description is given below with reference to the specific drawings and embodiments.

[0013] An embodiment of the present invention provides a smart diagnosis assistance method based on medical images, which may be applied to classify medical images. The present invention can efficiently process high-dimensional input data and at the same time ensure optimal classification performance, and thus is provided with strong practicability and extensibility. The present invention is applicable to all disease classification tasks that can be diagnosed with medical images, for the convenience of description, the present invention takes Alzheimer's disease as an example. Brain MRI (Magnetic Resonance Imaging) may be classified by this method, and the smart assistance diagnosis may be performed on the Alzheimer's disease according to classification results such as normal elderly, mild cognitive impairment and Alzheimer's disease. Specifically, based on a preset generator model, a preset discriminator model and a preset classifier model, sample images and classification categories corresponding to the sample images are trained to obtain a triple generative adversarial nets which includes a trained generator model, a trained discriminator model and a trained classifier model, and the MRI images are classified based on the trained classifier model to obtain corresponding classification results. Among them, the trained triple generative adversarial nets includes a tensor-decomposed network layer and a second-order pooling module.

[0014] In the present invention, the images are classified based on the triple generative adversarial nets, which, compared with a traditional binary generative adversarial network, alleviates the problem of unstable training of the traditional generative adversarial network and solves the problem of inconsistent convergence points of the discriminator in the traditional binary generative adversarial network when performing two target tasks of distinguishing true and false and classifying. In the present invention, when the images are classified, a tensorization method instead of a traditional vectorization method is used to compress each network layer in the triple generative adversarial nets, which achieves a regularization effect on the network model while reducing parameters, and solves the problems of excessive parameters and overfitting in classification and recognition of high-resolution images; further, the tensorization method can maintain internal correlation between the internal spatial structure information and different voxels, and solves the problem of internal structure loss of vectorized network layers. In the present invention, the traditional first-order pooling (maximum pooling or average pooling) is replaced by second-order pooling, and better discriminative features are automatically extracted under the action of the self-attention mechanism by making full use of the second-order information of the overall image, thereby improving the classification accuracy of the classifier. The triple generative adversarial nets in the present invention uses the semi-supervised learning method to jointly cooperate and train the preset generator model, the preset discriminator model and the preset classifier model, this reduces the network model's demand for image labeling information and makes full use of unlabeled data, thereby realizing the intelligent network model provided with high precision and high robustness; moreover, this reduces the number of parameters, improves the computing efficiency, helps reduce the performance requirements of the terminal, accelerates the speed of the medical image classification, and improves the disease diagnosis efficiency.

[0015] Please refer to FIG. 1. FIG. 1 is a schematic flowchart of a smart diagnosis assistance method based on medical images provided by an embodiment of the present invention. An execution subject of the smart assistance diagnosis method in this embodiment is a terminal, and the terminal includes but is not limited to a mobile terminal such as a smart phone, a tablet computer, and a PDA (Personal Digital Assistant etc., and may also include a terminal such as a desktop computer etc. As shown in FIG. 1, the following may be included in the smart assistance diagnosis method.

[0016] At S101, acquire a medical image to be classified.

[0017] When detecting a medical image classification instruction, the terminal acquires the medical image to be classified. The medical image classification instruction is an instruction for instructing the terminal to perform medical image classification. The image classification instruction may be triggered by a user, for example, a doctor clicks on an image classification option in the terminal. The acquired medical image to be classified may be a medical image to be classified that is uploaded by the user to the terminal, or may be a medical image to be classified that is extracted by the terminal from a text file, here the terminal acquires the text file corresponding to the file identifier based on a file identifier included in the image classification instruction.

[0018] At S102, preprocess the medical image to be classified to obtain a preprocessed image.

[0019] The terminal preprocesses the medical image to be classified to obtain the preprocessed image. Specifically, the terminal processes the medical image to be classified into single-color channel images, and stitches the single-color channel images to obtain the preprocessed image. A single-color channel image is a color channel image composed of information including one kind of color elements. The channel that saves the color information of the image is called the color channel, and each color channel stores the information of the color elements in the image. For example, in a RGB color mode, R represents a red channel, G represents a green channel, and B represents a blue channel. The terminal may convert the channel mode of the medical image to be classified into a plurality of single-color channel images by invoking a preset function; and stitch the plurality of single-color channel images by invoking the preset function to obtain the preprocessed image.

[0020] At S103, input the preprocessed image into a trained classification model for classification processing to obtain a classification category corresponding to the preprocessed image. Here the classification model includes a tensorized network layer and a second-order pooling module; the classification model is a triple generative adversarial nets obtained by training sample images and classification categories corresponding to the sample images based on a preset generator model, a preset discriminator model and a preset classifier model.

[0021] The terminal inputs the preprocessed image into the trained classification model for classification processing, and obtains the classification category corresponding to the preprocessed image. The trained classification model is the triple generative adversarial nets obtained by training the sample images and the classification categories corresponding to the sample images based on the preset generator model, the preset discriminator model and the preset classifier model; the trained classification model includes the tensor-decomposed network layer and the second-order pooling module.

[0022] During the training, the preset generator model, the preset discriminator model and the preset classifier model cooperate with each other and work together to train the training data to obtain a trained triple generative adversarial nets. Here, the training data may include the sample images, the classification categories corresponding to the sample images (that is, preset real image labels), and unlabeled sample images. Specifically, the terminal generates composite images provided with classification labels based on preset classification labels, one-dimensional Gaussian random vectors, and the preset generator model, and finally generates composite image label pairs; acquires sample image label pairs corresponding to the sample images through prediction based on the sample images and the preset classifier model; inputs the sample image label pairs, the preset real image label pairs and the composite image label pairs into the preset discriminator model for discrimination to obtain a first discrimination result corresponding to the sample image label pairs, a second discrimination result corresponding to the preset real image label pairs and a third discrimination result corresponding to the composite image label pairs; calculates a first loss function corresponding to the preset generator model, a second loss function corresponding to the preset discriminator model and a third loss function corresponding to the preset classifier model based on the first discrimination result, the second discrimination result and the third discrimination result; updates network parameters respectively corresponding to the preset generator model, the preset discriminator model and the preset classifier model through gradient descent of a back-propagation algorithm based on the first loss function, the second loss function and the third loss function; stops the training when the first loss function, the second loss function and the third loss function are all converged to obtain the triple generative adversarial nets, that is, to obtain the trained classification model, here the trained classification model includes a trained generator model, a trained discriminator model and a trained classifier model. The preprocessed image is input into the trained classifier model, and the trained classifier model normalizes the preprocessed image to obtain a target image; the trained classifier model is used to extract key features in the target image to acquire a global high-order feature map; the trained classifier model obtains the classification category corresponding to the global high-order feature map and outputs the classification category, that is, the classification category corresponding to the preprocessed image is obtained.

[0023] Further, in order to make the extracted global high-order feature map retain more important feature information, and thereby improving the classification accuracy of the classification model, the S103 may include S1031, S1032 and S1033 specifically as follows.

[0024] At S1031, normalize the preprocessed image by using the classifier model to obtain a target image.

[0025] The trained classification model includes a trained generator model, a trained discriminator model and a trained classifier model, and the preprocessed image may be classified through the trained classifier model. Specifically, the preprocessed image is input into the trained classifier model, and the preprocessed image is normalized to obtain the target image. For example, data corresponding to the preprocessed image is acquired, and the data is transformed linearly so that voxel values corresponding to the preprocessed image are between [-1, 1].

[0026] At S1032, extract key features in the target image by using the classifier model to obtain a global high-order feature map.

[0027] The terminal extracts the key features in the target image by using the trained classifier model to obtain the global high-order feature map; and the trained classifier model includes a tensor-decomposited network layer and a second-order pooling module. Please refer to FIG. 2. FIG. 2 is a structural schematic diagram of the trained classifier model provided by the present invention. As shown in FIG. 2, the trained classifier model includes a 3D convolution layer, a 3D average pooling layer, a 3D dense connection block 1, a second-order pooling module 1, a transition layer 1, a 3D dense connection block 2, a second-order pooling module 2, a transition layer 2, a 3D dense connection block 3, a second-order pooling module 3 and a full connection layer.

[0028] Specifically, during training the classification model, each network layer in the preset generator model, the preset discriminator model and the preset classifier model is tensorized. The classifier model is taken as an example, during the training, the network layers such as the 3D convolution layer, the 3D average pooling layer and the full connection layer etc. in the preset classifier model are tensorized. The terminal extracts the key features in the target image based on the tensor-decomposited network layers and the second-order pooling module in the trained classifier model to obtain the global high-order feature map.

[0029] In this embodiment, each network layer in the classifier model is tensorized, and each network layer in the classifier model is compressed by the tensorization method instead of the traditional vectorization method, which reduces parameters while achieving a regularization effect on the network model, thereby solving the problem of excessive parameters and overfitting in the classification and recognition of high-resolution images. In this embodiment, through replacing the traditional first-order pooling with the second-order pooling and taking full advantages of the second-order information of the input preprocessed image, and better discriminative features related to lesions are automatically extracted under the action of the self-attention mechanism, thereby improving the classification accuracy of the classifier; further, the classifier model uses the 3D convolution layer instead of a 2D convolution layer, so that the input image may be input in a form of tensor without any dimensionality reduction, thereby retaining the spatial information of the image and reducing the spatial information loss caused by the 2D network layer.

[0030] Further, in order to extract the global high-order feature map including more feature information, the S1032 may include S10321, S10322, S10323 and S10324 specifically as follows.

[0031] At S10321, extract features in the target image through the tensorized network layer in the classifier model to obtain a first feature map.

[0032] The terminal extracts the features in the target image through the tensorized network layer in the trained classifier model to obtain the first feature map. For example, the features in the target image are extracted through the 3D convolution layer, the 3D average pooling layer, etc. in the trained classifier model to obtain the first feature map.

[0033] At S10322, perform channel dimension reduction on the first feature map through the second-order pooling module in the classifier model to obtain a dimension-reduced second feature map.

[0034] The terminal performs the channel dimension reduction on the first feature map through the second-order pooling module in the trained classifier model to obtain the dimension-reduced second feature map. Please refer to FIG. 3. FIG. 3 is a structural schematic diagram of the second-order pooling module provided by the present invention, and the second-order pooling module includes a pre-shrinking module and a calibration module. Specifically, the channel dimension reduction is performed on the input four-dimensional feature map (i.e., the first feature map) through a 1×1×1 convolution to obtain the dimension-reduced second feature map.

[0035] At S10323, calculate a weight vector corresponding to the second feature map.

[0036] The terminal calculates the weight vector corresponding to the second feature map through the trained classifier model. Please refer to FIG. 3, specifically, covariance information of each two channels between different channels in the dimension-reduced second feature map is calculated to obtain a covariance matrix; and the weight vector having the same channel number with that of the four-dimensional feature map is acquired through a grouping convolution and the 1×1×1 convolution according to the covariance matrix.

[0037] At S10324, weight the first feature map based on the weight vector to obtain the global high-order feature map.

[0038] The terminal weights the first feature map based on the calculated weight vector, so that a weight of an important channel in the first feature map is larger and a weight of an unimportant channel is smaller, so as to obtain a more representative global high-order feature map. Specifically, the trained classifier model uses the back-propagation algorithm to make the weight of the important channel in the first feature map larger and the weight of the unimportant channel in the first feature map smaller, so as to extract more representative feature information and thus obtain the global high-order feature map.

[0039] At S1033, acquire the classification category corresponding to the global high-order feature map by using the classifier model.

[0040] The trained classifier model acquires the classification category corresponding to the global high-order feature map and outputs the classification category, that is, the classification category corresponding to the preprocessed image is acquired. Further, this classification category may be used to assist disease diagnosis in the embodiment of the present invention, and the medical image to be classified is obtained through the terminal; the medical image to be classified is preprocessed to obtain the preprocessed image; and the preprocessed image is classified based on the trained classification model to acquire the corresponding classification result. Since the trained classification model includes the tensor-decomposited network layer and the second-order pooling module, the internal structure information and the internal correlation of the medical image are preserved when the medical image is processed based on this classification model, and the second-order pooling module, based on the dependencies of different regions of the medical image and the correlation information between different channels of high-order features, makes the weight of the important feature channel larger and the weight of the unimportant feature channel smaller under the action of the self-attention mechanism, so as to extract better discriminative features associated with a lesion, thereby improving the accuracy of the smart assistance diagnosis for diseases. Further, in the present invention, the images are classified based on the triple generative adversarial nets, which, on the basis of the traditional binary generative adversarial network including a generator and a discriminator, adds up the classifier network model, and alleviates the problem of training instability of the binary generative adversarial network through designing a compatible loss function, solves the problem of inconsistent convergence points of the discriminator in the traditional binary generative adversarial network when performing two target tasks of distinguishing true and false and classifying, so that the generator model and the classifier model work in a cooperative training manner to replace the generative adversarial training method of the traditional binary generative adversarial network, thereby enabling the generator model and the classifier model to achieve the best optimality at the same time, speeding up the convergence, and making the generative adversarial network easier to reach the Nash equilibrium. In the present invention, when the medical images are classified, the tensorization method instead of the traditional vectorization method is used to compress each network layer in the triple generative adversarial nets, which achieves the regularization effect on the network model while reducing the parameters, and solves the problems of excessive parameters and overfitting in classification and recognition of high-resolution images; further, the tensorization method can maintain internal correlation between the internal spatial structure information and different voxels, and solves the problem of internal structure loss of the vectorized network layers. In the present invention, the traditional first-order pooling (maximum pooling or average pooling) is replaced by the second-order pooling, and the better discriminative features are automatically extracted under the action of the self-attention mechanism by making full use of the second-order information of the overall image, thereby improving the classification accuracy of the classifier. The triple generative adversarial nets in the present invention uses the semi-supervised learning method to jointly cooperate and train the preset generator model, the preset discriminator model and the preset classifier model, this reduces the network model's demand for image labeling information and makes full use of unlabeled data, thereby realizing the intelligent network model provided with high precision and high robustness; moreover, this reduces the number of the parameters, improves the computing efficiency, helps reduce the performance requirements of the terminal, thereby accelerating the assistance diagnosis process.

[0041] Please refer to FIG. 4. FIG. 4 is a schematic flowchart of a smart diagnosis assistance method based on medical images provided by another embodiment of the present invention. An execution subject of the smart assistance diagnosis method in this embodiment is a terminal, and the terminal includes but is not limited to a mobile terminal such as a smart phone, a tablet computer, a personal digital assistant etc., and may also include a terminal such as a desktop computer.

[0042] On the basis of the previous embodiment, training steps S201 and S202 of the classifier model is added into this embodiment. S203, S204 and S205 in this embodiment are exactly the same as the S101, S102 and S103 in the previous embodiment, the details of which may refer to the relevant descriptions for the S101, S102 and S103 in the previous embodiment and will not be repeated herein. In order to improve the accuracy of image classification, the S201 and S202 of the smart assistance diagnosis method as shown in FIG. 4 are as follows.

[0043] At S201, train sample images and classification categories corresponding to the sample images based on the preset generator model, the preset discriminator model and the preset classifier model to obtain the triple generative adversarial nets.

[0044] Please refer to FIG. 5. FIG. 5 is a structural schematic diagram of the triple generative adversarial nets provided by the present invention. Now, taking the invention scenario of the Alzheimer's disease as an example, the processes of training and acquiring the triple generative adversarial nets are illustrated by combining the structure of the triple generative adversarial nets in FIG. 5. During the training, the generator in FIG. 5 refers to the preset generator model, and the corresponding trained generator model is generated after the training is completed; during the training, the discriminator in FIG. 5 refers to the preset discriminator model, and the corresponding trained discriminator model is generated after the training is completed; during the training, the Alzheimer's disease classifier in FIG. 5 refers to the preset classifier model, and the corresponding trained classifier model is generated after the training is completed. The preset generator model mainly includes a 3D deconvolution layer; the preset discriminator model mainly includes a 3D convolution layer, a 3D dense connection block, a transition layer, a full connection layer, etc.; the preset classifier model mainly includes a 3D convolution layer, a 3D dense connection block, a second-order pooling module, etc. It is worth noting that the Alzheimer's disease is only taken as an example here to train and acquire the smart assistance diagnosis model of Alzheimer's disease that can be used to classify MRI images; but classification models used for other medical images may be acquired by training trained through this method, which is not limited herein.

[0045] During the training, each network layer in the preset generator model, the preset discriminator model and the preset classifier model is tensorized. Specifically, parameter compression is performed on the 3D convolution layer and the full connection layer in the preset discriminator model and the preset classifier model and on the 3D deconvolution layer in the preset generator model through the tensor decomposition method; and a weight matrix of the full connection layer, a convolution kernel tensor of the deconvolution layer and a convolution kernel tensor of the convolution layer may all be expressed in a corresponding tensor form: W i 1 j 1 , … , i d j d = G 1 i 1 j 1 G 2 i 2 j 2 … G d i d j d .

[0046] Tensor decomposition is performed on the weight tensor W of the full connection layer according to the above formula to obtain a tensorized representation of the full connection layer as follows: Y i 1 , … , j 1 = ∑ i 1 , … , j d G 1 i 1 j 1 G 2 i 2 j 2 … G d i d j d χ j 1 , … , j d + B i 1 , … , i d .

[0047] Steps of the tensor decomposition of the 3D convolution layer and the deconvolution layer are as follows: network input layer: χ x y z s → reshape χ ˜ x , y , z , s 1 , s 2 , … , s d , network output layer: Y x y z s → reshape Y ˜ x , y , z , s 1 , s 2 , … , s d , convolution kernel: K x y z c s → TT − format G 0 i j k G 1 c 1 s 1 … G d c d s d .

[0048] After the 3D convolution layer is tensorized, the following is obtained: Y ˜ x , y , z , s 1 , … , s d = ∑ i = 1 l ∑ j = 1 l ∑ k = 1 l ∑ c 1 , … , c d G 0 i j k G 1 c 1 s 1 … G d c d s d χ ˜ i + x − 1 , j + y − 1 , k + z − 1 , c 1 , … , c d where c = ∏ i = 1 d c i , s = ∏ i = 1 d s i .

[0049] During the training, the composite images provided with classification labels are generated based on the preset classification labels, the one-dimensional Gaussian random vectors and the preset generator model, and the composite image label pairs are finally generated; the sample image label pairs corresponding to the sample images are determined based on the sample images in the training data and the preset classifier model; the sample image label pairs, the preset real image label pairs and the composite image label pairs are input into the preset discriminator model for discrimination to obtain the first discrimination result corresponding to the sample image label pairs, the second discrimination result corresponding to the preset real image label pairs and the third discrimination result corresponding to the composite image label pairs; the first loss function corresponding to the preset generator model, the second loss function corresponding to the preset discriminator model and the third loss function corresponding to the preset classifier model are calculated based on the first discrimination result, the second discrimination result and the third discrimination result; the network parameters respectively corresponding to the preset generator model, the preset discriminator model and the preset classifier model are updated through gradient descent of the back-propagation algorithm based on the first loss function, the second loss function and the third loss function; and the training is stopped when the first loss function, the second loss function and the third loss function are all converged to obtain the triple generative adversarial nets, that is, to obtain the trained classification model. Here, the trained classification model includes the trained generator model, the trained discriminator model and the trained classifier model.

[0050] At S202, acquire the trained classifier model from the triple generative adversarial nets.

[0051] The trained classifier model is acquire from the triple generative adversarial nets. Specifically, the triple generative adversarial nets obtained by training is the trained classification model, and the trained classification model includes the trained generator model, the trained discriminator model and the trained classifier model, and the terminal acquires the trained classifier model from the trained classification model.

[0052] Further, in order to improve the accuracy of image classification and the speed of classification, in another embodiment, the S201 may include S2011, S2012, S2013, S2014, S2015 and S2016, which are specifically as follows.

[0053] At S2011, generate composite image label pairs based on preset classification labels, one-dimensional Gaussian random vectors, and the preset generator model.

[0054] Specifically, the one-dimensional Gaussian random vectors and the preset classification labels are used as input and are input into the preset generator model; the input preset classification labels are cascaded to each tensorized network layer through one-hot encoding, and the target feature map is generated based on the one-dimensional Gaussian random vectors; the target feature map is enlarged layer by layer based on the tensorized network layers, and the target composite image is finally generated; and thus the composite image label pairs are finally generated.

[0055] Further, in order to reduce the network model's demand for medical images, the semi-supervised learning method is introduced into the disease classification task, which can efficiently and comprehensively utilize the unlabeled medical image information, and at the same time the composite images provided with the classification labels are generated through the preset generator model, which plays a role of data enhancement and can train a high-precision assistance diagnosis model in a case of few number of samples. This reduces the demand for labeled training samples, reduces the workload of complicated labeling for training data in traditional algorithms, thereby further shortening the work period of disease diagnosis, accelerating the speed of disease diagnosis and improving the overall disease identification efficiency. The S2011 includes S20111, S20112 and S20113, which are as follows.

[0056] At S20111, cascade the preset classification labels to the tensorized network layers, and generate the target feature map based on the one-dimensional Gaussian random vectors.

[0057] The preset generator model includes tensorized network layers. For example, a tensorized deconvolution layer is included in the preset generator model. The input preset classification labels are cascaded to each tensorized deconvolution layer through the one-hot encoding, and the target feature map is generated based on the tensorized deconvolution layer and the one-dimensional Gaussian random vectors. Among them, an activation function of the deconvolution layer adopts a ReLU (Rectified Linear Unit) function and performs BN (Batch Normalization). In the assistance diagnosis and recognition of Alzheimer's disease, the target feature map is a anatomical brain feature map.

[0058] At S20112, enlarge the target feature map layer by layer based on the tensorized network layers to generate a target composite image.

[0059] After a plurality layers of deconvolution are performed, the tensorized network layers enlarge the target feature map layer by layer, and the obtained image is the target composite image. For example, the anatomical brain feature map is enlarged layer by layer to generate a composite image provided with the same size as a real MRI image. Here, a last layer of the preset generator model adopts a hyperbolic function, i.e., a tanh activation function.

[0060] At S20113, generate the composite image label pairs based on the target composite image and the preset classification labels.

[0061] The terminal generates the composite image label pairs based on the target composite image and the preset classification labels. For example, the terminal generates the MRI image provided with the classification label based on the composite image and the preset classification label. The MRI image provided with the classification label may also be referred to as an MRI image label pairs.

[0062] At S2012, predict sample image label pairs corresponding to the sample images based on the sample images and the preset classifier model.

[0063] The terminal predicts the corresponding category of the sample image based on the sample image in the training data and the preset classifier model, and determines the sample image label corresponding to the sample image based on the sample image and the corresponding category. Specifically, the sample image is input into the preset classifier model, the preset classifier model predicts the label information corresponding to the sample image and generates the sample image label pairs corresponding to the sample image based on the sample image and the label information. Taking the Alzheimer's disease as an example, the sample image is a real unlabeled MRI image, the real unlabeled MRI image is input into the preset classifier model, the preset classifier model predicts the label information corresponding to the real unlabeled MRI image and generates the MRI image label pairs based on the real unlabeled MRI image and the predicted label information.

[0064] At S2013, input the sample image label pair, the preset real image label pair, and the composite image label pair into the preset discriminator model for discrimination to obtain a first discrimination result corresponding to the sample image label pair, a second discrimination result corresponding to the preset real image label pair and a third discrimination result corresponding to the composite image label pair.

[0065] Specifically, the sample image label pair is input into the preset discriminator model for discrimination.

[0066] The preset discriminator model extracts the feature information of the sample image label pair to obtain the sample feature map corresponding to the sample image label pair, and the discrimination processing is performed on the sample feature map based on the preset discriminator model to obtain the first discrimination result. The preset real image label pair in the training data is input into the preset discriminator model for discrimination processing, and the preset discriminator model extracts the feature information of the preset real image label pair to obtain the real feature map corresponding to the preset real image label pair, and the discrimination processing is performed on the real feature map based on the preset discriminator model to obtain the second discrimination result. The composite image label pair is input into the preset discriminator model for discrimination processing, the preset discriminator model extracts the feature information of the composite image label pair to obtain the composite feature map corresponding to the composite image label pair, and the discrimination processing is performed on the composite feature map to obtain the third discriminant result.

[0067] Further, in order to improve the classification performance of the classification model and further improve the accuracy of the image classification, the S2013 may include S20131, S20132, S20133 and S20134, which are as follows.

[0068] At S20131, extract the feature information of the sample image label pair based on the tensorized dense convolutional neural network to obtain the sample feature map corresponding to the sample image label pair.

[0069] The preset discriminator model includes the tensorized dense convolutional neural network. The sample image corresponding to the sample image label pair is input into the preset discriminator model in a form of a third-order tensor, and the classification label in the sample image label pair, after one-hot encoded as a condition variable, is cascaded into each network layer of the preset discriminator model. Specifically, the tensorized dense convolutional neural network extracts the feature information of the sample image label pair to obtain the feature map that retains the spatial information, that is, to obtain the sample feature map corresponding to the sample image label pair. Here, the activation function of the convolution layer adopts ReLU and batch normalization.

[0070] At S20132, extract the feature information of the preset real image label pair based on the tensorized dense convolutional neural network to obtain the real feature map corresponding to the preset real sample image label pair.

[0071] The image corresponding to the real image label pair is input into the preset discriminator model in a form of a third-order tensor, and the classification label in the real image label pair, after one-hot encoded as a condition variable, is cascaded into each network layer of the preset discriminator model. Specifically, the tensorized dense convolutional neural network extracts the feature information of the real image label pair to obtain the feature map that retains the spatial information, that is, to obtain the real feature map corresponding to the real image label pair. Here, the activation function of the convolution layer adopts ReLU and batch normalization.

[0072] At S20133, extract the feature information in the composite image label pair based on the tensorized dense convolutional neural network to obtain the composite feature map corresponding to the composite image label pair.

[0073] The composite image is input into the preset discriminator model in a form of a third-order tensor, and the classification label corresponding to the composite image, after one-hot encoded as a condition variable, is cascaded into each network layer in the preset discriminator model. Specifically, the tensorized dense convolutional neural network extracts the feature information in the composite image label pair to obtain the feature map that retains the spatial information, that is, to obtain the composite feature map corresponding to the composite image label pair. Here, the activation function of the convolution layer adopts ReLU and batch normalization.

[0074] At S20134, perform discrimination processing on the sample feature map, the real feature map and the composite feature map respectively based on the preset discriminator model to obtain the first discrimination result, the second discrimination result and the third discrimination result.

[0075] Specifically, the preset discriminator model includes a tensorized full connection layer, and performs the discrimination based on a sigmoid function of the tensorized full connection layer to obtain the first discrimination result corresponding to the sample feature map, the second discrimination result corresponding to the real feature map and the third discrimination result corresponding to the composite feature map.

[0076] At S2014, calculate the first loss function corresponding to the preset generator model, the second loss function corresponding to the preset discriminator model and the third loss function corresponding to the preset classifier model based on the first discrimination result, the second discrimination result and the third discrimination result.

[0077] Specifically, during the process of training the preset generator model, the parameters of the tensor-decomposited kernel matrix G k [i k ,j k ] of the network layer in the preset generator model are updated in the backpropagation process according to the gradient descent of the loss function G-loss of the preset generator model. The goal of the preset generator model is to generate an image that can fool the preset discriminator model by simulating a real image. For example, the preset generator model generates a near-real MRI image that can fool the preset discriminator model by simulating real MRI anatomical brain structure features. Therefore, the loss of the preset generator model includes two parts, one part is to deceive the preset discriminator model to make the preset discriminator model discriminate the MRI image label pair generated by the preset generator model as true, and another part is reconstruction loss between the real MRI image and the generated MRI image, which may be expressed as: ∂ G − loss ∂ G k i k j k = ∇ θ g ∑ x g y g log 1 − D x g y g + λ x label − y g L 1 .

[0078] During the process of training the preset discriminator model, the parameters of the tensor-decomposited kernel matrix G k [i k ,j k ] of the network layer in the preset discriminator model are updated in the backpropagation process according to the gradient descent of the loss function G-loss of the preset discriminator model. The goal of the preset discriminator model is to discriminate the real image label pair as true and discriminate the composite image label pair generated by the preset generator model and provided with the classification label as well as the image label pair predicted by the preset classifier model as false. For example, the goal of the preset discriminator model is to discriminate the real MRI image label pair as true and to discriminate the MRI image label pair generated by the preset generator model and the MRI image label pair predicted by the preset classifier model as false. Therefore, the loss of the preset discriminator model includes three parts, which are specifically as follows: ∂ D − loss ∂ G k i k j k ︸ r k − 1 × r k = ∇ θ d ∑ x label y log D x label y + α ∑ x unlable y c log 1 − D x unlable y c + 1 − α ∑ x g y g log 1 − D x g y g .

[0079] During the process of training the preset classifier model, the parameters of the tensor-decomposited kernel matrix G k [i k ,j k ] of the network layer in the preset classifier model are updated in the backpropagation process according to the gradient descent of the loss function G-loss of the preset classifier model. The goal of the preset classifier model is to automatically extract and classify the feature information in the preprocessed image, for example, to automatically extract MRI anatomical brain structure features for classification and classify the MRI image into one of three categories: normal, Alzheimer's disease and mild cognitive impairment. Therefore, the loss of the preset classifier model includes two parts, one part is supervised loss, that is, cross-entropy of the classification tasks performed on the real image and the generated image; another part is unsupervised loss, that is, the preset discriminator model is deceived to make the preset discriminator model discriminate the MRI image label pair which is generated by the preset classifier model for the the unlabeled MRI image as true. Specifically, it may be expressed as: ∂ C − loss ∂ G k i k j k = ∇ θ c L sup ervised + L un sup ervised , L sup ervised = R L + α p R p , R L = E x label y ∼ P real x y − logP c y x label , R p = E x g y g ∼ P g x y − logP c y g x g , L un sup ervised = E x unlabel ∼ P c x log 1 − D x unlabel C x unlabel .

[0080] Calculation of R L for the real MRI image label pair is equivalent to calculation of KL divergence between distribution P c (x,y) learned by the preset classifier model and distribution P real (x,y) of the real data. The preset generator model generates the MRI image label pair close to the real distribution, which can improve the classification performance of the classifier model, therefore, R p is introduced to calculate the cross entropy of the generated MRI image label pair; minimization of R p is equivalent to minimization of the KL divergence D KL (P g (x,y)∥P c (x,y)); since D KL (P g (x,y)∥P c (x,y)) cannot be calculated directly, therefore the KL divergence D KL (P g (x,y)∥P c (x,y)) cannot be calculated directly either, and the preset classifier model achieves the purpose of minimizing the KL divergence D KL (P 8 (x,y)∥P c (x,y)) through indirectly minimizing R p .

[0081] At S2015, update network parameters respectively corresponding to the preset generator model, the preset discriminator model and the preset classifier model through gradient descent of a backpropagation algorithm respectively based on the first loss function, the second loss function and the third loss function.

[0082] The terminal updates the network parameters respectively corresponding to the preset generator model, the preset discriminator model and the preset classifier model through the gradient descent of the backpropagation algorithm respectively based on the calculated first loss function, second loss function and third loss function. For example, weight values etc. of each network layer in each of the preset generator model, the preset discriminator model and the preset classifier model are updated according to the first loss function, the second loss function and the third loss function. Then, the training is continued based on the parameter-updated preset generator model, preset discriminator model and preset classifier model. That is, the sample images and the classification categories corresponding to the sample images continue to be trained based on the respective parameter-updated models.

[0083] At S2016, stop training when the first loss function, the second loss function and the third loss function all converge, to obtain the triple generative adversarial nets.

[0084] The preset generator model, the preset discriminator model and the preset classifier model are cooperatively trained, such as repeatedly training in a cooperative mode of "generation-discrimination-classification". When the terminal detects that the first loss function, the second loss function and the third loss function have all converged during the training process that is in the repeated "generation-discrimination-classification" cooperative mode, the training is stopped, and then the trained triple generative adversarial nets is obtained, that is, the trained classification model. The number of iterations may also be preset, and it is considered that the training is completed when the training is performed up to this number of iterations, and the trained triple generative adversarial nets is obtained at this time.

[0085] In this embodiment of the present invention, the medical image to be classified is obtained through the terminal; the medical image to be classified is preprocessed to obtain the preprocessed image; and the preprocessed image is classified based on the trained classification model to acquire the corresponding classification result. Since the trained classification model includes the tensor-decomposited network layer and the second-order pooling module, the internal structure information and the internal correlation of the medical image are preserved when the medical image is processed based on this classification model, and the second-order pooling module, based on the dependencies of different regions of the medical image and the correlation information between different channels of high-order features, makes the weight of the important feature channel larger and the weight of the unimportant feature channel smaller under the action of the self-attention mechanism, so as to extract better discriminative features associated with a lesion, thereby improving the accuracy of the smart assistance diagnosis for diseases. Further, in the present invention, the images are classified based on the triple generative adversarial nets, which, on the basis of the traditional binary generative adversarial network including a generator and a discriminator, adds up the classifier network model, and alleviates the problem of training instability of the binary generative adversarial network through designing a compatible loss function, solves the problem of inconsistent convergence points of the discriminator in the traditional binary generative adversarial network when performing two target tasks of distinguishing true and false and classifying, so that the generator model and the classifier model work in a cooperative training manner to replace the generative adversarial training method of the traditional binary generative adversarial network, thereby enabling the generator model and the classifier model to achieve the best optimality at the same time, speeding up the convergence, and making the generative adversarial network easier to reach the Nash equilibrium. In the present invention, when the medical images are classified, the tensorization method instead of the traditional vectorization method is used to compress each network layer in the triple generative adversarial nets, which achieves the regularization effect on the network model while reducing the parameters, and solves the problems of excessive parameters and overfitting in classification and recognition of high-resolution images; further, the tensorization method can maintain internal correlation of the image between the internal spatial structure information and different voxels, and solves the problem of internal structure loss of the vectorized network layers. In the present invention, the traditional first-order pooling (maximum pooling or average pooling) is replaced by the second-order pooling, and the better discriminative features are automatically extracted under the action of the self-attention mechanism by making full use of the second-order information of the overall image, thereby improving the classification accuracy of the classifier. The triple generative adversarial nets in the present invention uses the semi-supervised learning method to jointly cooperate and train the preset generator model, the preset discriminator model and the preset classifier model, this reduces the network model's demand for image labeling information and makes full use of unlabeled data, thereby realizing the intelligent network model provided with high precision and high robustness; moreover, this reduces the number of the parameters, improves the computing efficiency, helps reduce the performance requirements of the terminal, thereby improving the efficiency of the smart assistance diagnosis.

[0086] Please refer to FIG. 6. FIG. 6 is a schematic diagram of a smart diagnosis assistance terminal based on medical images provided by an embodiment of the present invention. The units included in this terminal is used to execute the steps in the embodiments corresponding to FIG. 1 and FIG. 4 . For details, please refer to the relevant descriptions in the embodiments respectively corresponding to FIG. 1 and FIG. 4. For convenience of explanation, only the parts related to this embodiment are shown. Referring to FIG. 6, which includes: an acquisition unit 310 configured to acquire a medical image to be classified; a preprocessing unit 320 configured to preprocess the medical image to be classified to obtain a preprocessed image; a classification unit 330 configured to input the preprocessed image into a trained classification model for classification processing to obtain a classification category corresponding to the preprocessed image; here the classification model includes a tensorized network layer and a second-order pooling module, the classification model is a triple generative adversarial nets obtained by training sample images and classification categories corresponding to the sample images based on a preset generator model, a preset discriminator model and a preset classifier model.

[0087] The trained classification model includes a trained classifier model.

[0088] Further, the classification unit 330 includes: a processing unit configured to normalize the preprocessed image by using the classifier model to obtain a target image; an extraction unit configured to extract key features in the target image by using the classifier model to obtain a global high-order feature map; a classification category acquisition unit configured to acquire the classification category corresponding to the global high-order feature map by using the classifier model.

[0089] Further, the extraction unit is specifically configured to: extract features in the target image through the tensorized network layer in the classifier model to obtain a first feature map; perform channel dimension reduction on the first feature map through the second-order pooling module in the classifier model to obtain a dimension-reduced second feature map; calculate a weight vector corresponding to the second feature map; weight the first feature map based on the weight vector to obtain the global high-order feature map.

[0090] Further, the terminal further includes: a training unit configured to train the sample images and the classification categories corresponding to the sample images based on the preset generator model, the preset discriminator model and the preset classifier model to obtain the triple generative adversarial nets; a model acquisition unit configured to acquire the trained classifier model from the triple generative adversarial nets.

[0091] Further, the training unit includes: a generation unit configured to generate a composite image label pair based on preset classification labels, a one-dimensional Gaussian random vector and the preset generator model; a determination unit configured to predict a sample image label pair corresponding to the sample image based on the sample image and the preset classifier model; a discrimination unit configured to input the sample image label pair, a preset real image label pair and the composite image label pair into the preset discriminator model for discrimination processing to obtain a first discrimination result corresponding to the sample image label pair, a second discrimination result corresponding to the preset real image label pair and a third discrimination result corresponding to the composite image label pair; a calculation unit configured to calculate a first loss function corresponding to the preset generator model, a second loss function corresponding to the preset discriminator model and a third loss function corresponding to the preset classifier model based on the first discrimination result, the second discrimination result and the third discrimination result; an update unit configured to update network parameters respectively corresponding to the preset generator model, the preset discriminator model and the preset classifier model through gradient descent of a backpropagation algorithm respectively based on the first loss function, the second loss function and the third loss function; a network generation unit configured to stop training when the first loss function, the second loss function and the third loss function all converge to obtain the triple generative adversarial nets.

[0092] The preset generator model includes the tensorized network layer.

[0093] Further, the generation unit is specifically configured to: cascade the preset classification labels to the tensorized network layer, and generate the target feature map based on the one-dimensional Gaussian random vector; enlarge the target feature map layer by layer based on the tensorized network layer to generate a target composite image; generate the composite image label pair based on the target composite image and the preset classification labels.

[0094] The preset discriminator model includes a tensorized dense convolutional neural network.

[0095] Further, the discrimination unit is specifically configured to: extract feature information of the sample image label pair based on the tensorized dense convolutional neural network to obtain a sample feature map corresponding to the sample image label pair; extract feature information of the preset real image label pair based on the tensorized dense convolutional neural network to obtain a real feature map corresponding to the preset real sample image label pair; extract feature information in the composite image label pair based on the tensorized dense convolutional neural network to obtain a composite feature map corresponding to the composite image label pair; perform discrimination processing on the sample feature map, the real feature map and the composite feature map respectively based on the preset discriminator model to obtain the first discrimination result, the second discrimination result and the third discrimination result.

[0096] The present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the embodiments of the foregoing smart diagnosis assistance method.

Claims

1. A smart diagnosis assistance method based on medical images, comprising: acquiring a medical image to be classified; preprocessing the medical image to be classified to obtain a preprocessed image; inputting the preprocessed image into a trained classification model for classification processing to obtain a classification category corresponding to the preprocessed image; wherein the trained classification model is a triple generative adversarial nets obtained by training sample images and classification categories corresponding to the sample images based on a preset generator model, a preset discriminator model and a preset classifier model; wherein, the trained classification model comprises a trained classifier model, the trained classifier model comprises a tensorized network layer and a second-order pooling module; the inputting the preprocessed image into the trained classification model for classification processing to obtain the classification category corresponding to the preprocessed image comprises: normalizing the preprocessed image by using the classifier model to obtain a target image; extracting key features in the target image based on the tensorized network layer and the second-order pooling module of the trained classifier model to obtain a global high-order feature map; acquiring the classification category corresponding to the global high-order feature map by using the trained classifier model; the extracting the key features in the target image based on the tensorized network layer and the second-order pooling module of the trained classifier model to obtain the global high-order feature map comprises: extracting features in the target image through the tensorized network layer in the trained classifier model to obtain a first feature map; performing channel dimension reduction on the first feature map through the second-order pooling module in the trained classifier model to obtain a dimension-reduced second feature map; calculating a weight vector corresponding to the second feature map; and weighting the first feature map based on the weight vector to obtain the global high-order feature map; before acquiring the medical image to be classified, the method further comprises: training the sample images and the classification categories corresponding to the sample images based on the preset generator model, the preset discriminator model and the preset classifier model to obtain the triple generative adversarial nets; wherein each network layer in the preset generator model, the preset discriminator model and the preset classifier model is tensorized during a training process of the classification module; and acquiring the trained classifier model from the triple generative adversarial nets.

2. The smart diagnosis assistance method of claim 1, <b>characterized in that, the training the sample images and the classification categories corresponding to the sample images based on the preset generator model, the preset discriminator model and the preset classifier model to obtain the triple generative adversarial nets comprises: generating a composite image label pair based on preset classification labels, a one-dimensional Gaussian random vector and the preset generator model; predicting a sample image label pair corresponding to the sample image based on the sample image and the preset classifier model; inputting the sample image label pair, a preset real image label pair and the composite image label pair into the preset discriminator model for discrimination processing to obtain a first discrimination result corresponding to the sample image label pair, a second discrimination result corresponding to the preset real image label pair and a third discrimination result corresponding to the composite image label pair; calculating a first loss function corresponding to the preset generator model, a second loss function corresponding to the preset discriminator model and a third loss function corresponding to the preset classifier model based on the first discrimination result, the second discrimination result and the third discrimination result; updating network parameters respectively corresponding to the preset generator model, the preset discriminator model and the preset classifier model through gradient descent of a backpropagation algorithm respectively based on the first loss function, the second loss function and the third loss function; stopping training when the first loss function, the second loss function and the third loss function all converge to obtain the triple generative adversarial nets.

3. The smart diagnosis assistance method of claim 5, <b>characterized in that, the preset generator model comprises the tensorized network layer, and the generating the composite image label pair based on the preset classification labels, the one-dimensional Gaussian random vector and the preset generator model comprises: cascading the preset classification labels to the tensorized network layer, and generating the target feature map based on the one-dimensional Gaussian random vector; enlarging the target feature map layer by layer based on the tensorized network layer to generate a target composite image; generating the composite image label pair based on the target composite image and the preset classification labels.

4. The smart diagnosis assistance method of claim 2, <b>characterized in that, the preset discriminator model comprises a tensorized dense convolutional neural network; wherein the inputting the sample image label pair, the preset real image label pair and the composite image label pair into the preset discriminator model for discrimination processing to obtain the first discrimination result corresponding to the sample image label pair, the second discrimination result corresponding to the preset real image label pair and the third discrimination result corresponding to the composite image label pair comprises: extracting feature information of the sample image label pair based on the tensorized dense convolutional neural network to obtain a sample feature map corresponding to the sample image label pair; extracting feature information of the preset real image label pair based on the tensorized dense convolutional neural network to obtain a real feature map corresponding to the preset real sample image label pair; extracting feature information in the composite image label pair based on the tensorized dense convolutional neural network to obtain a composite feature map corresponding to the composite image label pair; performing discrimination processing on the sample feature map, the real feature map and the composite feature map respectively based on the preset discriminator model to obtain the first discrimination result, the second discrimination result and the third discrimination result.

5. A smart diagnosis assistance terminal based on medical images, comprising: an acquisition unit configured to acquire a medical image to be classified; a preprocessing unit configured to preprocess the medical image to be classified to obtain a preprocessed image; a classification unit configured to input the preprocessed image into a trained classification model for classification processing to obtain a classification category corresponding to the preprocessed image; wherein the trained classification model is a triple generative adversarial nets obtained by training sample images and classification categories corresponding to the sample images based on a preset generator model, a preset discriminator model and a preset classifier model; wherein, the trained classification model comprises a trained classifier model, the trained classifier model comprises a tensorized network layer and a second-order pooling module, and the classification unit comprises: a processing unit configured to normalize the preprocessed image by using the classifier model to obtain a target image; an extraction unit configured to extract key features in the target image based on the tensorized network layer and the second-order pooling module of the trained classifier model to obtain a global high-order feature map; a classification category acquisition unit configured to acquire the classification category; the extraction unit is specifically configured to: extract features in the target image through the tensorized network layer in the classifier model to obtain a first feature map; perform channel dimension reduction on the first feature map through the second-order pooling module in the classifier model to obtain a dimension-reduced second feature map; calculate a weight vector corresponding to the second feature map; weight the first feature map based on the weight vector to obtain the global high-order feature map; the terminal further comprises: a training unit configured to train the sample images and the classification categories corresponding to the sample images based on the preset generator model, the preset discriminator model and the preset classifier model to obtain the triple generative adversarial nets; wherein each network layer in the preset generator model, the preset discriminator model and the preset classifier model is tensorized during a training process of the classification module; and a model acquisition unit configured to acquire the trained classifier model from the triple generative adversarial nets.

6. The terminal of claim 5, characterized in that, the training unit comprises: a generation unit configured to generate a composite image label pair based on preset classification labels, a one-dimensional Gaussian random vector and the preset generator model; a determination unit configured to predict a sample image label pair corresponding to the sample image based on the sample image and the preset classifier model; a discrimination unit configured to input the sample image label pair, a preset real image label pair and the composite image label pair into the preset discriminator model for discrimination processing to obtain a first discrimination result corresponding to the sample image label pair, a second discrimination result corresponding to the preset real image label pair and a third discrimination result corresponding to the composite image label pair; a calculation unit configured to calculate a first loss function corresponding to the preset generator model, a second loss function corresponding to the preset discriminator model and a third loss function corresponding to the preset classifier model based on the first discrimination result, the second discrimination result and the third discrimination result; an update unit configured to update network parameters respectively corresponding to the preset generator model, the preset discriminator model and the preset classifier model through gradient descent of a backpropagation algorithm respectively based on the first loss function, the second loss function and the third loss function; a network generation unit configured to stop training when the first loss function, the second loss function and the third loss function all converge to obtain the triple generative adversarial nets.

7. The terminal of claim 6, <b>characterized in that, the generation unit is specifically configured to: cascade the preset classification labels to the tensorized network layer, and generate the target feature map based on the one-dimensional Gaussian random vector; enlarge the target feature map layer by layer based on the tensorized network layer to generate a target composite image; generate the composite image label pair based on the target composite image and the preset classification labels.

8. The terminal of claim 6, characterized in that, the discrimination unit is specifically configured to: extract feature information of the sample image label pair based on the tensorized dense convolutional neural network to obtain a sample feature map corresponding to the sample image label pair; extract feature information of the preset real image label pair based on the tensorized dense convolutional neural network to obtain a real feature map corresponding to the preset real sample image label pair; extract feature information in the composite image label pair based on the tensorized dense convolutional neural network to obtain a composite feature map corresponding to the composite image label pair; perform discrimination processing on the sample feature map, the real feature map and the composite feature map respectively based on the preset discriminator model to obtain the first discrimination result, the second discrimination result and the third discrimination result.

9. A computer-readable storage medium, on which a computer program is stored, characterized in that, the computer program, when executed by a processor, implements the smart diagnosis assistance method of any one of claims 1-4.