Image Processing Method, Apparatus, Computer Product, and Storage Medium

Abstract

Embodiments of the present application disclose an image processing method, apparatus, computer product, and storage medium. The image processing method includes: transforming a first image to obtain multiple second images; selecting a first machine learning module from a group including at least one first machine learning module to perform feature extraction on the second images to obtain a feature map corresponding to each second image; and inputting the feature map corresponding to each second image into a second machine learning module to obtain a processing result of the first image. According to the technical solution of the embodiments of the present application, the accuracy and robustness of image processing are effectively improved.

Description

Technical Field

[0001] This application generally relates to the field of computer technology, and more particularly to an image processing method and apparatus, a computer product, and a storage medium. Background Art

[0002] Image processing technology has a wide range of applications in many fields such as geodetic images, remote sensing, and medical images. Therefore, how to quickly and accurately process image data has always been one of the hot topics in the computer field.

[0003] With the development of machine learning technology, more and more machine learning models and algorithms have been introduced into image processing technology. Currently, most of the commonly used image processing models are single models with simple structures, so there are problems of low accuracy and poor robustness in image recognition and processing. Summary of the Invention

[0004] In a first aspect, an embodiment of this application provides an image processing method, including:

[0005] Transforming a first image to obtain a plurality of second images;

[0006] Selecting a first machine learning module from a group including at least one first machine learning module to extract features from the second images, obtaining a feature map corresponding to each second image; and

[0007] Inputting the feature map corresponding to each second image into a second machine learning module to obtain a processing result of the first image.

[0008] In a second aspect, an embodiment of this application further provides an image processing apparatus, including:

[0009] An image transformation module configured to transform a first image to obtain a plurality of second images;

[0010] A feature extraction module including at least one first machine learning module, configured to extract features from the second images through the first machine learning module to obtain a feature map corresponding to each second image; and

[0011] An image processing module including a second machine learning module, configured to process the feature map corresponding to each second image to obtain a processing result of the first image.

[0012] In a third aspect, an embodiment of this application further provides a computer product, including:

[0013] One or more processors;

[0014] The one or more processors are configured to execute the image processing method provided by each embodiment of this application when running computer instructions.

[0015] In a fourth aspect, embodiments of the present application further provide a computer-readable storage medium storing computer instructions run by a processor, and when the computer instructions are run by the processor, they can execute the image processing method provided by the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objects, and advantages of the present application will become more apparent:

[0017] Figure 1 Shows an exemplary flowchart of the image processing method according to an embodiment of the present application;

[0018] Figure 2 Shows an exemplary flowchart of the process of feature extraction of the second image in the image processing method according to an embodiment of the present application;

[0019] Figure 3 Shows an exemplary flowchart of the process of image processing by the second machine learning module in the image processing method according to an embodiment of the present application;

[0020] Figure 4 Shows a schematic structural diagram of the image processing apparatus provided by an embodiment of the present application; and

[0021] Figure 5 Shows a schematic structural diagram of a computer product for implementing the method of the embodiment of the present application according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The present application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related invention and not to limit the invention. Additionally, it should be noted that for the sake of description, only the parts related to the invention are shown in the drawings.

[0023] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and embodiments.

[0024] Unless otherwise defined, technical terms or scientific terms used in this disclosure shall have the ordinary meanings as understood by those of ordinary skill in the art to which this invention pertains. The "first", "second" and similar terms used in this disclosure do not denote any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "comprising" or "including" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. Words such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections or signal connections, whether direct or indirect.

[0025] In the following embodiments, the communication connection includes connection via a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The network may include a local area network, the Internet, a telecommunications network, the Internet of Things based on the Internet and / or a telecommunications network, and / or any combination of the above networks, etc. The wired network may, for example, use metal traces, twisted pairs, coaxial cables, or optical fiber transmission, etc. for information transmission. The wireless network may, for example, use communication methods such as a WWAN mobile communication network, Bluetooth, Zigbee, or Wi-Fi.

[0026] In the following embodiments, the processor is a processor device having the ability to perform logical operations, such as a central processing unit (CPU), a field programmable gate array (FPGA), a digital signal processor (DSP), a microcontroller unit (MCU), an application specific integrated circuit (ASIC), a graphics processing unit (GPU), etc., which have the ability to process data and / or execute programs. It is easy to understand that the processor is usually communicatively connected to a memory, and any combination of one or more computer program products is stored on the memory. The memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may, for example, include random access memory (RAM) and / or cache memory, etc. Non-volatile memory may, for example, include read-only memory (ROM), a hard disk, erasable programmable read-only memory (EPROM), a USB memory, a flash memory, etc. One or more computer instructions may be stored on the memory, and the processor may run the computer instructions to implement related analysis functions. Various application programs and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the application programs, etc.

[0027] In the following embodiments, each module can be implemented by a processor executing relevant computer instructions. For example, the image transformation module can be implemented by a processor executing instructions of an image transformation algorithm, the machine learning module can be implemented by a processor executing instructions of a machine learning algorithm, and the neural network can be implemented by a processor executing instructions of a neural network algorithm.

[0028] In the following embodiments, each module can run on the same processor or on multiple processors; each module can run on processors of the same architecture, for example, all running on processors of the X86 architecture, or can run on processors of different architectures, for example, the image transformation module runs on a CPU of the X86 architecture, and the first and second machine learning modules run on a GPU. Each module can be encapsulated in a computer product. For example, each module is encapsulated in a computer software and runs on a computer (server), or can be individually or partially encapsulated in different computer products. For example, the image transformation module is encapsulated in a computer software and runs on a computer (server), and the first and second machine learning modules are respectively encapsulated in separate computer softwares and run on another or multiple computers (servers); the computing platform during the execution of each module can be local computing, cloud computing, or a hybrid computing composed of local computing and cloud computing.

[0029] In the following embodiments, the implementation of machine learning (such as a neural network) can adopt an existing machine learning (such as a neural network) framework with pre-set parameters, or can adopt an existing machine learning (such as a neural network) framework to obtain the required parameters through training with relevant image samples of the embodiments of the present application, or can also be to establish a machine learning (such as a neural network) framework according to the essential spirit of the invention of the embodiments of the present application and conduct training.

[0030] Figure 1 An exemplary flowchart of an image processing method according to an embodiment of the present application is shown.

[0031] As Figure 1 shown, the image processing method provided by the embodiment of the present application includes:

[0032] Step S10: Transform the first image to obtain multiple second images.

[0033] In the embodiment of the present application, multiple second images can be obtained by transforming the first image. This transformation process can adopt different transformation methods according to different information contained in the first image, or can also adopt different transformation methods according to different features to be extracted.

[0034] Optionally, step S10 includes:

[0035] Transform the first image according to the transformation rules to obtain multiple second images.

[0036] In the embodiments of the present application, the transformation rules may vary according to the different features of the first image or according to the different features to be extracted. For the first image, it can be processed according to one or more transformation rules.

[0037] Optionally, the transformation rules include at least one of spatial domain transformation, value domain transformation or time domain transformation, and can also be based on the above transformation rules to select the transformation recommended by the professional field related to the first image.

[0038] Among them, spatial domain transformation, such as rotating, flipping, translating, stretching, distorting, deforming, scaling the image, etc.; value domain transformation, such as image filtering, image segmentation, image occlusion, image compression, chromaticity mapping, colorization, grayscale conversion, etc.; time domain transformation, such as autocorrelation function transformation, cross-correlation function transformation, etc.

[0039] Optionally, when the first image is an image with specific applications or properties, in order not to affect the deep essential features contained in the first image, the transformation performed adopts the transformation recommended by the professional field related to the first image.

[0040] For example, if the first image is a medical image, the transformation performed is a transformation that does not change the pathological properties of the medical image, including spatial domain transformations usually adopted in medical image transformation, such as rotating, flipping, translating, stretching, distorting, deforming, scaling the image, etc.; and color transformations, such as grayscale conversion, binarization of the image, etc.

[0041] Optionally, when transforming the first image, multiple second images can be obtained based on a single transformation rule, or multiple second images can be obtained based on multiple transformation rules.

[0042] For example, taking the dermoscopic image of melanoma as an example, 1 image can be transformed by rotation with different degrees to obtain 5 second images; or 1 image can be rotated to 3 images and processed based on the color constancy theory to obtain 2 images, resulting in 5 second images; or 1 image can be both rotated and processed based on the color constancy theory to be transformed into 5 second images.

[0043] For example, taking the dermoscopic image of melanoma as an example, under the guidance of professionals in the medical image field such as doctors, 1 original dermoscopic image can be transformed into 5 second images by performing transformations such as rotation, flipping, and grayscale conversion on it.

[0044] In the above embodiments, by transforming and processing the first image, without affecting the effective information of the image, the amount of image data is increased, which helps to improve the accuracy of image processing and recognition.

[0045] Step S20: Select a first machine learning module from a group including at least one first machine learning module, and perform feature extraction on the second image to obtain multiple feature maps corresponding to each second image.

[0046] In the embodiments of the present application, the first machine learning module can be implemented by using any available feature extraction algorithm in the field of machine learning. For example, principal component analysis, Fisher linear discriminant, projection pursuit, neural network, genetic algorithm, etc. The features to be extracted can be various features included in the image. For example, including but not limited to color, grayscale, texture, contour, region, special points, lines, and any combination of these features, etc.

[0047] In the above embodiments, selecting a suitable first machine learning module for feature extraction according to requirements can effectively remove the useless features and noise interference in the first image, prevent the mutual interference when different second images are subjected to feature extraction together, so as to retain the essential features of the first image in each second image, and improve the accuracy and adaptability of the image processing process.

[0048] Optionally, the first machine learning module includes a first neural network.

[0049] In some embodiments of the present application, in the group including at least one first machine learning module, each first neural network is the same. The relevant training can be performed in advance according to the training process of the general neural network to obtain different parameters, and the first neural network with the best performance effect is selected for feature extraction.

[0050] In some embodiments of the present application, in the group including at least one first machine learning module, at least some of the first neural networks of the first machine learning modules are different. For example, the architectures are the same but the parameters are different, or the architectures are different. The relevant training can be performed in advance according to the training process of the general neural network, and one or several first neural networks with the best performance effect are selected for feature extraction. When using, when performing feature extraction on each second image, a first neural network is randomly selected from one or several first neural networks with the best performance effect for feature extraction.

[0051] In some embodiments of the present application, in a group including at least one first machine learning module, the first neural network of at least part of the first machine learning modules is selected by those skilled in the art from the existing neural networks in the art for at least part of the second images suitable for feature extraction for the type of the transformed second images, and these first neural networks are used one-to-one for the feature extraction of the targeted second images; alternatively, optionally, multiple first neural networks with different architectures are trained, and part of the first neural networks formed by training are selected to perform feature extraction one-to-one with at least part of the second images; alternatively, optionally, multiple first neural networks with the same architecture are trained, and part of the first neural networks with different parameters formed by training are selected to perform feature extraction one-to-one with at least part of the second images.

[0052] Among them, the first neural network can adopt various common neural network architectures such as convolutional neural network CNN, deep neural network DNN, recurrent neural network RNN, autoencoder neural network AENN, sparse autoencoder SAE, etc., or adopt neural network products implemented based on common neural network architectures, such as AlexNet, VGGNet, Google Inception Net, ResNet, etc., and can also design a neural network structure according to the principle of the neural network.

[0053] Step S30: Input the feature maps corresponding to each second image into the second machine learning module to obtain the processing result of the first image.

[0054] In the embodiments of the present application, all the feature maps corresponding to the second images obtained in step S20 are input into the second machine learning module for processing, and this processing can be one or more image processing processes, such as image analysis, segmentation, recognition, classification, clustering, super-resolution improvement, retrieval, etc.

[0055] In some embodiments of the present application, the image processing method is to classify or cluster the first image.

[0056] In the embodiments of the present application, the second machine learning module can be any algorithm for processing feature maps, such as support vector machine, simple fusion, co-training fusion, neural network, etc.

[0057] In some embodiments of the present application, the second machine learning module includes a second neural network. The second neural network can adopt various common neural network architectures such as convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), autoencoder neural network (AENN), sparse autoencoder (SAE), etc., or adopt neural network products implemented based on common neural network architectures, such as AlexNet, VGGNet, Google Inception Net, ResNet, etc. It can also design a neural network structure according to the principle of neural networks.

[0058] It is easy for those skilled in the art to understand that, for the need of image processing purposes, the second neural network has its corresponding structure. For example, when the image processing is to classify / cluster an image, a classifier is provided in the output layer of the second neural network; when the image processing is to improve the resolution of an image, a deconvolution layer is provided in the second neural network.

[0059] In the above embodiments, by inputting each transformed second image into the first machine learning module for feature map extraction and inputting the feature map into the second machine learning module for processing, the accuracy and robustness of image processing are greatly improved.

[0060] Figure 2 An exemplary flowchart showing the process of feature extraction of the second image in the image processing method according to an embodiment of the present application is shown.

[0061] As Figure 2 shown, in one embodiment, step S20 includes:

[0062] Step S21: A group including at least one first machine learning module includes a plurality of first machine learning modules.

[0063] In some embodiments of the present application, each first machine learning module is pre-trained according to the training process of general machine learning, and a plurality of machine learning modules with better performance are selected as candidates for feature extraction. This performance can be measured according to the performance of the machine learning module on the test set.

[0064] Step S22: Select a first machine learning module from the plurality of first machine learning modules, for example, randomly select, or select in combination with the experience of those skilled in the art, and extract features from one of the second images, and repeat this process until the feature extraction of all second images is completed.

[0065] For each of the multiple second images, feature extraction can be performed from the above-selected first machine learning modules. Each image in the second images can use the same first machine learning module for feature extraction, or different first machine learning modules for feature extraction.

[0066] The different first machine learning modules mentioned herein may have the same architecture but different design parameters, or may have different architectures.

[0067] That is to say, a trained and effective first machine learning module can be selected to extract features from each second image, or for some second images, first machine learning modules suitable for them can be separately selected for feature extraction, or for each second image, a corresponding first machine learning module can be separately selected for feature extraction.

[0068] In some embodiments of the present application, for each second image transformed from the first image, a first machine learning module suitable for the corresponding transformation rule is selected to extract the features of the corresponding second image to improve the processing accuracy. For example, for a second image generated by grayscale transformation, a first machine learning module based on the convolutional neural network CNN is used for feature extraction; for a second image generated by transformations such as stretching and distortion, a first machine learning module based on the deep neural network DNN is used for feature extraction; for a second image generated by transformations such as randomly adding image noise, a first machine learning module based on the denoising autoencoder DAE is used for feature extraction.

[0069] In some embodiments of the present application, it further includes step S23: the selected first machine learning module extracts features from the second image and outputs feature maps corresponding to each second image with the same resolution. For example, taking the first machine learning module implemented by the first neural network as an example, feature maps corresponding to each second image with the same resolution are output through at least one intermediate layer of the first neural network of the selected first machine learning module.

[0070] By such a method, the pressure on the second machine learning module during image processing is effectively reduced.

[0071] For example, for a dermoscopic image, according to its image features, the Inception v3 neural network can be selected, and the 5 second images obtained after transforming the dermoscopic image are all subjected to feature extraction through the Inception v3 neural network. Each second image is input into the Inception v3 model. Since the intermediate layer of this model can extract 768 feature maps of 17*17 and can also extract 1280 feature maps of 8*8, according to the features of the dermoscopic image to be extracted, feature maps are selected. For example, 1280 feature maps with a resolution of 8*8 are selected, and a total of 5*1280 feature maps are extracted.

[0072] For example, for dermoscopic images, five second images obtained after transforming the dermoscopic images are used. Among them, the grayscale and binary second images are subjected to feature extraction through the Inception v3 neural network, and the rotated, stretched, and flipped second images are subjected to feature extraction through the Inception-ResNet-v2 neural network. Different intermediate layers of the two neural networks are respectively selected so that both output 768 feature maps of 17*17, and a total of 5*768 feature maps are extracted.

[0073] Those skilled in the art can understand that based on the rapid development of deep learning neural networks and the rapid growth of hardware computing capabilities, it is also possible to select the feature maps of some second images with different resolutions for output. For example, the resolution of some feature maps is 8*8, the resolution of some feature maps is 16*16, and the resolution of some feature maps is 32*32.

[0074] In the above embodiments, through the first machine learning module, such as the first neural network, feature map extraction is performed, which greatly improves the robustness and fault tolerance of the image processing process; at the same time, by selecting a suitable first machine learning module according to requirements to screen and extract feature maps, the image processing speed, accuracy, and stability are effectively optimized.

[0075] Figure 3 The exemplary flowchart of the image processing process by the second machine learning module in the image processing method according to an embodiment of the present application is shown.

[0076] As Figure 3 shown, in some embodiments of the present application, step S30 includes:

[0077] Step S31: Obtain the second machine learning module.

[0078] For example, an existing neural network can be selected or a new neural network can be generated through training as the second machine learning module.

[0079] In some embodiments of the present application, the second machine learning module can be the same as or different from a certain first machine learning module in a group including at least one first machine learning module, and the specific network structure can be pre-constructed and trained according to specific requirements.

[0080] For example, taking the use of the second neural network as the second machine learning module as an example, the structure of the second neural network can be designed as a convolutional layer - fully connected layer - LR classifier to classify the first image, or can be designed as a convolutional layer - pooling layer - fully connected layer - SVM classifier to classify the first image, or can also be designed as a convolutional layer - pooling layer - K - means clustering to cluster the first image, or can be designed as a convolutional layer - fully connected layer - deconvolutional layer to enhance the resolution of the first image, and so on. Specifically, it can be designed according to the needs of the image processing purpose.

[0081] Step S32: Input multiple feature maps corresponding to each second image into the second machine learning module.

[0082] The object processed by the second machine learning module is all the feature maps obtained in step S20. The pixels, dimensions, or quantities of these feature maps can be the same or different.

[0083] Step S33: The second machine learning module outputs the processing result of the first image.

[0084] The second machine learning module can fuse all the feature maps together, process them, obtain the final processing result and output it.

[0085] In the above - mentioned embodiment, by inputting all multiple feature maps into the second machine learning module for processing, the accuracy and robustness of image processing are improved.

[0086] Figure 4 The structural schematic diagram of an image processing device provided by an embodiment of the present application is shown.

[0087] As Figure 4 shown, the image processing device includes:

[0088] Image transformation module 10: Configured to transform the first image to obtain multiple second images.

[0089] In the embodiment of the present application, each first image can be transformed by the image transformation module 10 to obtain multiple second images. This transformation process can adopt different transformation techniques according to different information contained in the first image, or can also adopt different transformation techniques according to different features to be extracted.

[0090] In some embodiments of the present application, the image transformation unit 10 is configured to: process the first image according to the transformation rule to obtain multiple second images.

[0091] In some embodiments of the present application, the transformation rules of the image transformation module 10 can be different according to the different features of the first image, or can be different according to the different features to be extracted. For each first image, it can be processed according to one or more transformation rules.

[0092] For example, for a dermoscopic image of melanoma, one image can be only rotated to obtain five second images; one image can be rotated into three images and processed into two images based on the color constancy theory to obtain five second images; one image can also be rotated and processed based on the color constancy theory to transform and obtain five second images.

[0093] Optionally, the transformation rules include spatial domain transformation, value domain transformation or time domain transformation, and it is also possible to select a transformation recommended by the professional field related to the first image based on the above transformation rules.

[0094] Among them, spatial domain transformation, such as rotating, flipping, translating, stretching, distorting, deforming, scaling an image, etc.; value domain transformation, such as image filtering, image segmentation, image occlusion, image compression, chromaticity mapping, colorization, grayscale conversion, etc.; time domain transformation, such as autocorrelation function transformation, cross-correlation function transformation, etc.

[0095] Optionally, when the first image is an image with specific applications or properties, in order not to affect the deep essential features contained in the first image, the transformation performed adopts the transformation recommended by the professional field related to the first image.

[0096] For example, when the first image is a medical image, the transformation performed is a transformation that does not change the pathological properties of the medical image, including spatial domain transformations usually adopted in medical image transformation, such as rotating, flipping, translating, stretching, distorting, deforming, scaling an image, etc.; and color transformations, such as grayscale conversion, binarization of an image, etc.

[0097] For example, for a dermoscopic image of melanoma, under the guidance of a doctor, the image transformation module 10 can transform one original dermoscopic image into five second images by rotating and flipping it.

[0098] In the above embodiments, by transforming the first image through the image transformation module, the amount of image data is increased on the basis of not affecting the effective information of the image, which helps to improve the accuracy of image processing and recognition in subsequent processing.

[0099] Feature extraction module 20: includes at least one first machine learning module, configured to select the first machine learning module to extract features from the second image to obtain a feature map corresponding to each second image.

[0100] In the embodiments of the present application, the first machine learning module may be implemented by any available feature extraction algorithm in the field of machine learning. For example, but not limited to, principal component analysis, Fisher linear discriminant, projection pursuit, neural network, genetic algorithm, etc. The features to be extracted may be various features included in the image. For example, but not limited to, color, grayscale, texture, contour, region, special points, lines, and any combination of these features, etc.

[0101] In the above embodiments, selecting a suitable first machine learning module for feature extraction according to requirements can effectively remove the useless features and noise interference in the first image, prevent the mutual interference when different second images are subjected to feature extraction together, so as to retain the essential features of the first image in each second image, thereby improving the accuracy and adaptability of the image processing process.

[0102] Optionally, the first machine learning module includes a first neural network.

[0103] In some embodiments of the present application, each first neural network is the same. The relevant training can be carried out in advance according to the training process of a general neural network to obtain different parameters, and the first neural network with the best performance effect is selected for feature extraction.

[0104] In some embodiments of the present application, at least some of the first neural networks are different. For example, they may have the same architecture but different parameters, or may have different architectures. The relevant training can be carried out in advance according to the training process of a general neural network, and one or several first neural networks with the best performance effect are selected for feature extraction. When in use, when each second image is subjected to feature extraction, one is randomly selected from one or several first neural networks with the best performance effect for feature extraction.

[0105] In some embodiments of the present application, at least some of the first neural networks are selected by those skilled in the art from the existing neural networks in the field for at least part of the second images suitable for feature extraction according to the types of the transformed second images, and these first neural networks are used for the feature extraction of the corresponding second images one by one; or, optionally, multiple first neural networks with different architectures are trained, and some of the trained first neural networks are selected to correspond to at least part of the second images one by one for feature extraction; or, optionally, multiple first neural networks with the same architecture are trained, and some of the trained first neural networks with different parameters are selected to correspond to at least part of the second images one by one for feature extraction.

[0106] Among them, the first neural network can adopt various common neural network architectures such as convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), autoencoder neural network (AENN), sparse autoencoder (SAE), etc., or adopt neural network products implemented based on common neural network architectures, such as AlexNet, VGGNet, Google Inception Net, ResNet, etc. It can also design a neural network structure according to the principle of the neural network.

[0107] Optionally, the feature extraction module 20 is configured to:

[0108] Select a first machine learning module to extract features from one of the multiple second images, and repeat this process until the feature extraction of all second images is completed.

[0109] In some embodiments of the present application, the feature extraction module 20 includes multiple first machine learning modules and is specifically configured to

[0110] First, each first machine learning module is pre-trained, and the first machine learning module with better performance is selected as a candidate for feature extraction. This performance can be measured according to its performance on the test set, and the evaluation index depends on specific requirements.

[0111] Then, for each of the multiple second images, a first machine learning module can be randomly selected from the above-mentioned candidate first machine learning modules for feature extraction. Each of the second images can use the same first machine learning module for feature extraction, or different first machine learning modules for feature extraction.

[0112] In the feature extraction process of some embodiments of the present application, the feature extraction module 20 is configured such that the first machine learning module extracts features from the second image and outputs a feature map corresponding to each second image with the same resolution. For example, taking the first machine learning module implemented by the first neural network as an example, a feature map corresponding to each second image with the same resolution is output through at least one intermediate layer of the first neural network of the selected first machine learning module..

[0113] For example, for dermoscopic images, according to their image features, the Inception v3 neural network is selected, and the five second images obtained after transforming the dermoscopic images are all subjected to feature extraction through the Inception v3 neural network. Each second image is input into the Inception v3 model. Since the middle layer of this model can extract 768 feature maps of 17*17 and can also extract 1280 feature maps of 8*8, according to the features of the dermoscopic images to be extracted, feature maps are selected. For example, 1280 feature maps with a resolution of 8*8 are selected for extraction, and a total of 5*1280 feature maps are extracted.

[0114] For example, for dermoscopic images, among the five second images obtained after transforming the dermoscopic images, the second images that are gray-scaled and binaryzied are subjected to feature extraction through the Inception v3 neural network, and the second images that are rotated, stretched, and flipped are subjected to feature extraction through the Inception-ResNet-v2 neural network. Different middle layers of the two neural networks are respectively selected so that they both output 768 feature maps of 17*17, and a total of 5*768 feature maps are extracted.

[0115] Those skilled in the art can understand that based on the rapid development of deep learning neural networks and the rapid growth of hardware computing power, the feature extraction module 20 can also be configured to output feature maps of partial second images with different resolutions. For example, the resolution of some feature maps is 8*8, the resolution of some feature maps is 16*16, and the resolution of some feature maps is 32*32.

[0116] In the above embodiments, by using the first machine learning module to extract the feature maps of the second images adapted to the first image transformation, the robustness and fault tolerance of the image processing process are improved; at the same time, by selecting a suitable first machine learning module and the resolution of the feature maps output by it according to the requirements, the image processing speed, accuracy, and stability are effectively optimized.

[0117] The image processing module 30 includes a second machine learning module: configured to input all the feature maps corresponding to each second image into the second machine learning module to obtain the processing result of the first image.

[0118] In the embodiments of the present application, the image processing module 30 is used to input all the feature maps obtained in the feature extraction module 20 into the second machine learning module for processing. This processing can be one or more image processing processes, such as image analysis, segmentation, recognition, classification, clustering, or retrieval, etc. The second machine learning module can be any algorithm or model for feature map fusion, such as support vector machine, simple fusion, co-training fusion, neural network, etc.

[0119] In some embodiments of the present application, in the image processing module 30, the object processed by the second machine learning module is all the obtained feature maps, and the pixels, sizes or quantities of the feature maps may be the same or different. The second machine learning module can fuse all the feature maps together, analyze and process them, and obtain and output the final processing result.

[0120] In some embodiments of the present application, the second machine learning module includes a second neural network, such as a neural network known in the related art or a neural network obtained by training a known neural network to generate new parameters or forming a new architecture.

[0121] In some embodiments of the present application, the second machine learning module can adopt one of the first machine learning modules, or can be designed, constructed and trained according to the needs of the image processing to be completed to obtain a new structure.

[0122] For example, in some embodiments of the present disclosure, taking the use of the second neural network as the second machine learning module as an example, the second neural network structure can be designed as a convolutional layer - fully connected layer - LR classifier to classify the first image, or can be designed as a convolutional layer - pooling layer - fully connected layer - SVM classifier to classify the first image, or can also be designed as a convolutional layer - pooling layer - K - means clustering to cluster the first image, or can be designed as a convolutional layer - fully connected layer - deconvolutional layer to improve the resolution of the first image, and so on. Specifically, it can be designed according to the needs of the image processing purpose.

[0123] In the above - mentioned embodiments, the second machine learning module in the image processing module 30 processes multiple feature maps output by the feature extraction module, improving the accuracy and robustness of image processing.

[0124] It should be noted that although the connection relationships between the image transformation module, the feature extraction module, and the image processing module are not specifically described above, those skilled in the art can understand that the image transformation module and the feature extraction module are communicatively connected, and the feature extraction module and the image processing module are communicatively connected based on the technical solutions described in the above - mentioned embodiments.

[0125] In some embodiments of the present application, a computer product is further provided, including: one or more processors; the one or more processors are configured to execute the image processing method provided according to the above - mentioned various embodiments when running a computer program.

[0126] In some embodiments of the present application, as Figure 5As shown, an implementation of a computer product in terms of its structure is provided. Computer product 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage section 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of device 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0127] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read from it can be installed into the storage section 508 as needed.

[0128] Specifically, according to an embodiment of the present disclosure, the process described above with reference to Figure 1 can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly embodied on a machine-readable medium, and the computer program includes program code for performing Figure 1 the method. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 509, and / or installed from the removable medium 511.

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0130] The units or modules involved in the embodiments described in this application can be implemented in software or in hardware. The described units or modules can also be provided in a processor. The names of these units or modules do not, in some cases, constitute a limitation on the units or modules themselves.

[0131] In some embodiments, a computer-readable storage medium is also provided. The computer-readable storage medium can be the computer-readable storage medium included in the computer product described in the above embodiments, or can exist separately and be not assembled into the computer product. The provided computer-readable storage medium stores computer instructions run by a processor, and when the computer instructions are run by the processor, they can execute the image processing method provided according to the embodiments of the present application.

[0132] The above description is only a preferred embodiment of the present application and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, technical solutions formed by mutually replacing the above features with (but not limited to) technical features having similar functions disclosed in the present application.

Claims

1. An image processing method, characterized in that, The method includes: Transforming a first image according to a transformation rule to obtain multiple second images, where the transformation rule includes at least one of spatial domain transformation, value domain transformation, or time domain transformation. The first image includes a medical image, and the transformation of the first image according to the transformation rule is a transformation that does not change the pathological nature of the medical image. Specifically, it includes: adopting different transformation methods for the first image to obtain multiple different second images; among them, the spatial domain transformation includes one or more of image rotation, flipping, translation, stretching, distortion, deformation, and scaling; the value domain transformation includes one or more of image filtering, image segmentation, image occlusion, image compression, chromaticity mapping, colorization, and grayscale conversion; the time domain transformation includes one or more of autocorrelation function transformation and cross-correlation function transformation; Selecting a first machine learning module corresponding to the transformation rule from a group including at least one first machine learning module, and performing feature extraction on each of the second images to obtain a feature map corresponding to each of the second images, including: Selecting a first machine learning module corresponding to the transformation rule from a group including at least one first machine learning module, and performing feature extraction on each of the second images; repeating the step until the feature extraction of the multiple second images is completed to obtain all the feature maps corresponding to the multiple second images; among them, in the group including at least one first machine learning module, the first neural networks of at least some of the first machine learning modules are different; the selection of a first machine learning module corresponding to the transformation rule from a group including at least one first machine learning module includes at least one or more of the following methods: For the second image generated based on grayscale conversion, a first machine learning module based on a convolutional neural network (CNN) is used for feature extraction; For the second image generated based on at least one of stretching and distortion, a first machine learning module based on a deep neural network (DNN) is used for feature extraction; For the second image generated by randomly adding image noise transformation, a first machine learning module based on a denoising autoencoder (DAE) is used for feature extraction; When using the selected first machine learning module to perform feature extraction on the multiple second images, when performing feature extraction on each of the second images, a first neural network with the best performance is randomly selected from one or several first neural networks for feature extraction; and feature maps corresponding to each of the second images with the same output resolution are output; Performing feature extraction on each of the second images through at least one intermediate layer of the first neural network of the selected first machine learning module, and outputting feature maps corresponding to each of the second images with the same output resolution, where the first machine learning module includes a first neural network; Inputting the multiple feature maps corresponding to each of the second images into a second machine learning module for classification or clustering processing to obtain the processing result of the first image.

2. The method according to claim 1, characterized in that, Selecting a first machine learning module corresponding to the transformation rule from a group including at least one first machine learning module, and performing feature extraction on each of the second images to obtain a feature map corresponding to each of the second images, includes: For at least some of the second images, selecting a first machine learning module corresponding one-to-one to the at least some of the second images from a group including at least one first machine learning module to perform feature extraction, and obtaining a feature map corresponding to each of the second images.

3. The method according to claim 1, wherein The second machine learning module includes a second neural network.

4. An image processing apparatus, characterized in that, Includes: An image transformation module, configured to transform a first image according to a transformation rule, the transformation rule including at least one of a spatial domain transformation, a value domain transformation, or a time domain transformation, the first image including a medical image, and the transformation of the first image according to the transformation rule is a transformation that does not change the pathological nature of the medical image, specifically including: using different transformation methods for the first image to obtain multiple different second images; wherein, the spatial domain transformation includes one or more of image rotation, flipping, translation, stretching, distortion, deformation, scaling, etc., the value domain transformation includes one or more of image filtering, image segmentation, image occlusion, image compression, chromaticity mapping, colorization, grayscale conversion, etc., and the time domain transformation includes one or more of autocorrelation function transformation, cross-correlation function transformation, etc.; A feature extraction module, configured to select a first machine learning module corresponding to the transformation rule from a group including at least one first machine learning module, and perform feature extraction on each of the second images to obtain a feature map corresponding to each of the second images, includes: Selecting a first machine learning module corresponding to the transformation rule from a group including at least one first machine learning module, and performing feature extraction on each of the second images; repeating the steps until the feature extraction of the multiple second images is completed to obtain all the feature maps corresponding to the multiple second images; and wherein, in the group including at least one first machine learning module, the first neural networks of at least some of the first machine learning modules are different; the selection of a first machine learning module corresponding to the transformation rule from a group including at least one first machine learning module includes at least one or more of the following methods: For the second images generated based on grayscale conversion, using a first machine learning module based on a convolutional neural network (CNN) for feature extraction; For the second images generated based on at least one of stretching and distortion transformations, using a first machine learning module based on a deep neural network (DNN) for feature extraction; For the second images generated by randomly adding image noise transformation, using a first machine learning module based on a denoising autoencoder (DAE) for feature extraction; When using the selected first machine learning module to perform feature extraction on each of the second images, randomly selecting one from one or several first neural networks with the best performance effect for each of the second images to extract features when performing feature extraction on each of the second images; outputting feature maps corresponding to each of the second images with the same output resolution; The first machine learning module is configured to perform feature extraction on each of the second images through at least one intermediate layer of the first neural network of the selected first machine learning module, and output a feature map corresponding to each of the second images with the same resolution; and An image processing module, including a second machine learning module, is configured to input the feature maps corresponding to multiple second images into the second machine learning module for classification or clustering processing to obtain a processing result of the first image.

5. The image processing apparatus according to claim 4, wherein The feature extraction module is configured to, for at least some of the second images, select a first machine learning module corresponding to each of the at least some of the second images from a group including at least one first machine learning module to perform feature extraction, and obtain a feature map corresponding to each of the second images.

6. The image processing apparatus according to claim 4, wherein The second machine learning module includes a second neural network.

7. A computer product, characterized in that, Comprising: One or more processors; The one or more processors are configured to execute the method according to any one of claims 1-3 when running computer instructions.

8. A computer-readable storage medium storing computer instructions run by a processor, the computer instructions being used to execute the method according to any one of claims 1-3 when run by the processor.

Images