Neural network model training method and device, hyperspectral image classification method and device and electronic equipment

Abstract

The invention relates to a neural network model training method and device, a hyperspectral image classification method and device and electronic equipment, and the neural network model training comprises the steps: inputting a sample hyperspectral image into an initial image classification model, and determining a spectral feature map of the sample hyperspectral image under at least one spatial scale, obtaining an initial image classification result corresponding to the sample hyperspectral image; determining comparison loss corresponding to the initial image classification model according to the spectral feature map; determining classification loss corresponding to the initial image classification model according to the initial image classification result and a sample label corresponding to the sample hyperspectral image; and training the initial image classification model according to the comparison loss and the classification loss, and determining a trained target image classification model. Through the target image classification model provided by the embodiment of the invention, pixel-level image classification can be carried out on the hyperspectral image, high accuracy and reliability are achieved, and the sensitivity and specificity requirements of classification of the hyperspectral image are met.

Description

Technical Field

[0001] The present disclosure relates to the technical field of hyperspectral image processing, and in particular to a neural network model training, a hyperspectral image classification method and device, and an electronic device. Background Art

[0002] In treatments such as cancer treatment that require the removal of diseased tissue, it is very important to accurately identify diseased tissue and healthy tissue through medical images and determine the edge of the diseased tissue. In the prior art, the classification of diseased tissue and healthy tissue is usually based on traditional medical images. However, the sensitivity and specificity of the commonly used image classification methods in the prior art are low, which will affect the accuracy of the identification of the edge of the diseased tissue. Summary of the invention

[0003] In view of this, the present disclosure proposes a technical solution for neural network model training, a hyperspectral image classification method and device, and an electronic device.

[0004] According to one aspect of the present disclosure, a neural network model training method is provided, comprising: inputting a sample hyperspectral image into an initial image classification model, determining a spectral feature map of the sample hyperspectral image at at least one spatial scale, and an initial image classification result corresponding to the sample hyperspectral image; determining a contrast loss corresponding to the initial image classification model based on the spectral feature map, wherein the contrast loss is used to reflect the degree of similarity between spectral features corresponding to any two pixels in the sample hyperspectral image; determining a classification loss corresponding to the initial image classification model based on the initial image classification result and a sample label corresponding to the sample hyperspectral image; training the initial image classification model based on the contrast loss and the classification loss, and determining a trained target image classification model, wherein the target image classification model is used to perform pixel-level image classification of lesion areas and normal areas in medical hyperspectral images.

[0005] In a possible implementation, determining the contrast loss corresponding to the initial image classification model according to the spectral feature map includes: unifying the spatial scale of the spectral feature map to determine an adjusted spectral feature map, wherein the spatial scale corresponding to the adjusted spectral feature map is the same as the spatial scale corresponding to the sample hyperspectral image; determining the spectral feature corresponding to each pixel in the sample hyperspectral image according to the adjusted spectral feature map; and determining the contrast loss according to the spectral feature corresponding to each pixel in the sample hyperspectral image.

[0006] In a possible implementation, determining the contrast loss according to the spectral features corresponding to each pixel in the sample hyperspectral image includes: classifying the pixels in the sample hyperspectral image according to the sample label, and determining the classification result corresponding to each pixel, wherein the classification result corresponding to any pixel is used to indicate whether the pixel belongs to the lesion area; determining the contrast loss according to the classification result and spectral features corresponding to each pixel.

[0007] In a possible implementation, the initial image classification model is trained according to the contrast loss and the classification loss to determine a trained target image classification model, including: according to a preset loss weight, summing the contrast loss and the classification loss to determine the target loss corresponding to the initial image classification model; according to the target loss, the initial image classification model is trained to determine the target image classification model.

[0008] In a possible implementation, the method further includes: performing a performance evaluation on the target image classification model to determine a performance evaluation result corresponding to the target image classification model; adjusting the loss weight according to the performance evaluation result to determine an adjusted loss weight, wherein the adjusted loss weight is used to retrain the target image classification model.

[0009] According to another aspect of the present disclosure, a hyperspectral image classification method is provided, comprising: acquiring a medical hyperspectral image corresponding to a target lesion area; performing pixel-level image classification on the medical hyperspectral image using a target image classification model, and determining a target classification result corresponding to the medical hyperspectral image, wherein the target image classification model is trained by the above-mentioned neural network model training method, and the target classification result is used to indicate a lesion area and a normal area in the medical hyperspectral image.

[0010] According to another aspect of the present disclosure, a neural network model training device is provided, comprising: an image processing module, used to input a sample hyperspectral image into an initial image classification model, determine a spectral feature map of the sample hyperspectral image at at least one spatial scale, and an initial image classification result corresponding to the sample hyperspectral image; a first loss determination module, used to determine a contrast loss corresponding to the initial image classification model based on the spectral feature map, wherein the contrast loss is used to reflect the degree of similarity between spectral features corresponding to any two pixels in the sample hyperspectral image; a second loss determination module, used to determine a classification loss corresponding to the initial image classification model based on the initial image classification result and a sample label corresponding to the sample hyperspectral image; a model training module, used to train the initial image classification model based on the contrast loss and the classification loss, and determine a trained target image classification model, wherein the target image classification model is used to perform pixel-level image classification of lesion areas and normal areas in medical hyperspectral images.

[0011] According to another aspect of the present disclosure, a hyperspectral image classification device is provided, comprising: an image acquisition module, used to acquire a medical hyperspectral image corresponding to a target lesion area; an image classification module, used to perform pixel-level image classification on the medical hyperspectral image using a target image classification model, and determine a target classification result corresponding to the medical hyperspectral image, wherein the target image classification model is trained by the above-mentioned neural network model training method, and the target classification result is used to indicate a lesion area and a normal area in the medical hyperspectral image.

[0012] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0013] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0014] In the disclosed embodiment, by inputting the sample hyperspectral image into the initial image classification model, the spectral feature map of the sample hyperspectral image at at least one spatial scale and the initial image classification result corresponding to the sample hyperspectral image can be determined, and then the contrast loss that can reflect the similarity between the spectral features corresponding to any two pixels in the sample hyperspectral image can be determined according to the spectral feature map, so that the contrast learning of global semantic information in the pixel dimension can be realized through the spectral information in the sample hyperspectral image; according to the initial image classification result and the sample label corresponding to the sample hyperspectral image, the classification loss corresponding to the initial image classification model can be determined to provide a classification basis from the image dimension; according to the contrast loss and the classification loss, the initial image classification model is trained to determine the trained target image classification model, so that the target image classification model can make full use of the spectral information in the sample hyperspectral image, perform pixel-level image classification on the lesion area and the normal area in the medical hyperspectral image, improve the accuracy and reliability of the target image classification model, and meet the sensitivity and specificity requirements of medical image classification.

[0015] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0017] Figure 1 A flowchart showing a neural network model training method according to an embodiment of the present disclosure is shown;

[0018] Figure 2 A schematic diagram showing the structure of an initial image classification model according to an embodiment of the present disclosure is shown;

[0019] Figure 3 A box plot of a target image classification model according to an embodiment of the present disclosure is shown;

[0020] Figure 4 A flow chart showing a hyperspectral image classification method according to an embodiment of the present disclosure is shown;

[0021] Figure 5 A block diagram of a neural network model training device according to an embodiment of the present disclosure is shown;

[0022] Figure 6 A block diagram of a hyperspectral image classification device according to an embodiment of the present disclosure is shown;

[0023] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0025] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0026] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0027] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0028] In the prior art, the treatment plan for liver cancer and other cancers is still mainly surgical resection, that is, the diseased tissue is removed along the edge of the diseased tissue and normal tissue through surgery. In the process of removing the diseased tissue, on the one hand, it is necessary to remove all the diseased tissue as much as possible, and on the other hand, it is necessary to retain healthy tissue as much as possible to ensure that the corresponding organs or tissues can maintain their basic physiological functions, that is, it is necessary to reduce the residual rate of diseased tissue and increase the retention rate of healthy tissue. Therefore, how to accurately distinguish between diseased tissue and healthy tissue and determine the edge of the diseased tissue is an important part of the treatment plan with surgical resection as the core.

[0029] When using any image classification method to classify medical images for diseased tissues and healthy tissues, the accuracy (ACC), sensitivity (SEN) and specificity (SPE) corresponding to the image classification method can be calculated according to the specific content of each classification result and whether each classification result is accurate, so as to comprehensively describe the accuracy of the image classification method. Among them, the accuracy can generally describe the accuracy of the image classification method for diseased tissues and healthy tissues; the sensitivity can reflect the detection rate for diseased tissues; and the specificity can reflect the detection rate for healthy tissues.

[0030] Referring to Table 1, for any classification result, the classification result can be compared with the actual pathological detection result, and the classification result is defined as true positive (True Positive, TP), false positive (False Positive, FP), true negative (True Negative, TN) and false negative (False Negative, FN).

[0031] Table 1

[0032]

[0033] Based on this, the accuracy of image classification can be expressed as formula (1):

[0034]

[0035] Among them, TP represents the number of true positive classification results; TN represents the number of true negative classification results; FP represents the number of false positive classification results; FN represents the number of false negative classification results.

[0036] The sensitivity of image classification can be expressed as formula (2):

[0037]

[0038] The specificity of image classification can be expressed as formula (3):

[0039]

[0040] Image classification methods in the prior art are usually based on traditional medical images to perform image classification of diseased tissue and healthy tissue. The sensitivity and specificity of these image classification methods are usually low, that is, the detection rate of diseased tissue and the detection rate of healthy tissue are reduced, which will affect the accuracy and reliability of surgical resection based on its classification results, increase the residual rate of diseased tissue, and reduce the retention rate of healthy tissue.

[0041] In view of this, the present disclosure provides a neural network model training method, which can realize the contrast learning of global semantic information in the pixel dimension through the spectral information in the sample hyperspectral image, determine the contrast loss, and train the initial image classification model in combination with the classification loss in the image dimension, and determine the trained target image classification model, so that the target image classification model can make full use of the spectral information in the sample hyperspectral image, perform pixel-level image classification on the lesion area and normal area in the medical hyperspectral image, improve the accuracy and reliability of the target image classification model, and meet the sensitivity and specificity requirements of medical image classification. The neural network model training method provided by the embodiment of the present disclosure is described in detail below.

[0042] Figure 1 A flowchart of a neural network model training method according to an embodiment of the present disclosure is shown. The neural network model training method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The neural network model training method can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the neural network model training method can be executed by a server. Figure 1 As shown, the neural network model training method includes:

[0043] In step S11, the sample hyperspectral image is input into an initial image classification model to determine a spectral feature map of the sample hyperspectral image at at least one spatial scale and an initial image classification result corresponding to the sample hyperspectral image.

[0044] The sample hyperspectral image here can represent an image obtained by collecting images of any area including diseased tissue using spectral imaging technology. The specific form of the diseased tissue can be flexibly set according to actual usage requirements, for example, it can include tumor tissue, endoproliferative tissue, etc., and the present disclosure does not make specific limitations on this. Preferably, the sample hyperspectral image is a medical hyperspectral image corresponding to liver tumor tissue.

[0045] The specific number of sample hyperspectral images can be flexibly set according to actual usage requirements, and the present disclosure does not make specific limitations on this. The specific form of any sample hyperspectral image can refer to the implementation methods in the relevant technology, and the present disclosure does not make specific limitations on this.

[0046] The specific form of the initial image classification model can refer to the implementation methods in the relevant technology. For example, it can be a fully convolutional neural network (U-net) for biomedical image segmentation, a deep convolutional neural network (DeepLab series) for semantic image segmentation, etc. The present disclosure does not make specific limitations on this.

[0047] Using the initial image classification model, the sample hyperspectral image can be preliminarily classified to determine the corresponding initial image classification results to indicate the predicted lesion area and the predicted normal area in the sample hyperspectral image. The lesion area can represent the area corresponding to the lesion tissue in the sample hyperspectral image, and the normal area can represent the area corresponding to the healthy tissue in the sample hyperspectral image. The specific form of the initial image classification result can be flexibly set according to actual usage requirements. For example, it can include image segmentation results corresponding to the lesion area and image segmentation results corresponding to the normal area, etc. This disclosure does not make specific limitations on this.

[0048] In the process of the initial image classification model classifying the sample hyperspectral image, a spectral feature map of the sample hyperspectral image at at least one spatial scale can be extracted from the initial image classification model. The spectral feature map at any spatial scale can include spectral information of the sample hyperspectral image at the spatial scale, and its specific form can refer to the implementation in the related art, and the present disclosure does not specifically limit this.

[0049] The specific method of determining the spectral characteristic diagram of the sample hyperspectral image at at least one spatial scale can be flexibly set according to actual usage requirements and depends on the specific form of the initial image classification model. The present disclosure does not make specific limitations on this.

[0050] In a possible implementation, a branch structure for outputting a spectral feature map can be added at each network layer for feature extraction in the initial image classification model to determine the spectral feature map of the sample hyperspectral image at at least one spatial scale. The specific form of the branch structure can be flexibly set according to actual usage requirements and depends on the specific form of the initial image classification model, which is not specifically limited in this disclosure.

[0051] Figure 2 FIG. 2 shows a schematic diagram of the structure of an initial image classification model according to an embodiment of the present disclosure. Figure 2As shown in the figure, the initial image classification model is a U-net neural network model. The encoder of the U-net neural network model can extract the spectral feature graphs of the sample hyperspectral image at three spatial scales through three different convolutional layers. By adding a branch structure at each convolutional layer in the encoder, the spectral feature graph extracted by the convolutional layer can be obtained without affecting the original network structure of the U-net neural network model, so as to be used for subsequent comparative learning of spectral features. The U-net neural network model with the added branch structure can also be called a U-net neural network model based on contrastive learning (Contrastive Learning U-net, CL-unet).

[0052] Among them, the specific number of branch structures and the specific position of each branch structure in the initial image classification model can be flexibly set according to actual usage requirements and depend on the specific form of the initial image classification model. This disclosure does not make specific limitations on this.

[0053] Taking the U-net neural network model as an example, the branch structure can be set at the convolution layer of the encoder of the U-net neural network model as well as the convolution layer of the decoder of the U-net neural network model.

[0054] In step S12, the contrast loss corresponding to the initial image classification model is determined according to the spectral feature map, wherein the contrast loss is used to reflect the similarity between the spectral features corresponding to any two pixels in the sample hyperspectral image.

[0055] When high-dimensional spectral features are mapped to a specific low-dimensional space, the distance between spectral features of the same type in the low-dimensional space is closer, and the distance between spectral features of different types in the low-dimensional space is farther, that is, the distance within the class is reduced and the distance between classes is increased. Based on this idea, the spectral feature map of the sample hyperspectral image at at least one spatial scale can be used to implement comparative learning on the initial image classification model based on the spectral features.

[0056] Specifically, the spectral feature corresponding to any pixel in the sample hyperspectral image can be determined based on the spectral feature map, and then the spectral feature corresponding to each pixel can be used to compare the spectral features pixel by pixel, or randomly select pixels, analyze the similarity between the spectral features corresponding to any two pixels, perform comparative learning of global semantic information in the pixel dimension, and determine the contrast loss corresponding to the initial image classification model. It should be noted that the any two pixels mentioned here are not limited to two different pixels from the same sample hyperspectral image, but can also be two different pixels from different sample hyperspectral images.

[0057] The process of determining the contrast loss corresponding to the initial image classification model according to the spectral feature map will be described in detail later in conjunction with possible implementation methods of the present disclosure, which will not be elaborated here.

[0058] In step S13, the classification loss corresponding to the initial image classification model is determined according to the initial image classification result and the sample label corresponding to the sample hyperspectral image.

[0059] Among them, the sample label corresponding to the sample hyperspectral image is used to indicate the real diseased area and the real normal area in the sample hyperspectral image. Its specific form can be flexibly set according to actual usage requirements, and the present disclosure does not make specific limitations on this.

[0060] According to the initial image classification results and sample labels, the classification loss corresponding to the initial image classification model can be determined to reflect the classification ability of the initial image classification model for the sample highlight image. Among them, the specific form of the classification loss can refer to the implementation method in the relevant technology, and the present disclosure does not make specific limitations on this. Preferably, the classification loss corresponding to the initial image classification model can be set to a cross entropy loss. The specific expression form of the cross entropy loss can refer to the implementation method in the relevant technology, and the present disclosure does not make specific limitations on this.

[0061] In step S14, the initial image classification model is trained according to the contrast loss and the classification loss to determine the trained target image classification model, wherein the target image classification model is used to perform pixel-level image classification of lesion areas and normal areas in medical hyperspectral images.

[0062] According to the contrast loss and classification loss, the initial image classification model can be trained to improve the model performance of the initial image classification model from the pixel dimension and the image dimension, so that the trained target image classification model can classify the lesion area and the normal area in the medical hyperspectral image at the pixel level and improve the classification accuracy. The specific method of training the initial image classification model can refer to the implementation method in the related art, and the present disclosure does not make specific limitations on this.

[0063] In the disclosed embodiment, by inputting the sample hyperspectral image into the initial image classification model, the spectral feature map of the sample hyperspectral image at at least one spatial scale and the initial image classification result corresponding to the sample hyperspectral image can be determined, and then the contrast loss that can reflect the similarity between the spectral features corresponding to any two pixels in the sample hyperspectral image can be determined according to the spectral feature map, so that the contrast learning of global semantic information in the pixel dimension can be realized through the spectral information in the sample hyperspectral image; according to the initial image classification result and the sample label corresponding to the sample hyperspectral image, the classification loss corresponding to the initial image classification model can be determined to provide a classification basis from the image dimension; according to the contrast loss and the classification loss, the initial image classification model is trained to determine the trained target image classification model, so that the target image classification model can make full use of the spectral information in the sample hyperspectral image, perform pixel-level image classification on the lesion area and the normal area in the medical hyperspectral image, improve the accuracy and reliability of the target image classification model, and meet the sensitivity and specificity requirements of medical image classification.

[0064] In a possible implementation, the contrast loss corresponding to the initial image classification model is determined according to the spectral feature map, including: unifying the spatial scale of the spectral feature map to determine an adjusted spectral feature map, wherein the spatial scale corresponding to the adjusted spectral feature map is the same as the spatial scale corresponding to the sample hyperspectral image; determining the spectral feature corresponding to each pixel in the sample hyperspectral image according to the adjusted spectral feature map; and determining the contrast loss according to the spectral feature corresponding to each pixel in the sample hyperspectral image.

[0065] Since the spatial scale of the spectral feature map may be different from that of the sample hyperspectral image, in order to ensure the accuracy and reliability of the spectral features of each pixel, the spatial scale of the spectral feature map can be unified so that the spatial scale corresponding to the adjusted spectral feature map is the same as the spatial scale corresponding to the sample hyperspectral image. The specific method of unifying the spatial scale of the spectral feature map can refer to the specific method in the related art, and the present disclosure does not make specific limitations on this.

[0066] With the above Figure 2 For example, Figure 2 As shown, after inputting any sample hyperspectral image into the U-net neural network model, the spectral feature map with a spatial scale of 256, the spectral feature map with a spatial scale of 512, and the spectral feature map with a spatial scale of 1024 can be extracted at the three convolutional layers in the encoder of the U-net neural network model respectively. Through the branch structure set at each convolutional layer, the spatial scale of each spectral feature map can be unified to obtain three adjusted spectral feature maps with a spatial scale of 256.

[0067] According to the adjusted spectral feature graph, the spectral feature corresponding to each pixel in the sample hyperspectral image is determined, and the spectral feature corresponding to each pixel in the sample hyperspectral image can be determined. Among them, the specific form of the spectral feature corresponding to any pixel can be flexibly set according to actual usage requirements, for example, it can be set to a feature vector, etc., and the present disclosure does not make specific limitations on this. The specific method of determining the spectral feature corresponding to each pixel can refer to the implementation method in the relevant technology, and the present disclosure does not make specific limitations on this.

[0068] According to the spectral features corresponding to each pixel in the sample hyperspectral image, the spectral features can be compared and analyzed pixel by pixel to determine the contrast loss.

[0069] In a possible implementation, the contrast loss is determined according to the spectral features corresponding to each pixel in the sample hyperspectral image, including: classifying the pixels in the sample hyperspectral image according to the sample label, and determining the classification result corresponding to each pixel, wherein the classification result corresponding to any pixel is used to indicate whether the pixel belongs to the lesion area; and determining the contrast loss according to the classification result and the spectral features corresponding to each pixel.

[0070] According to the sample label corresponding to the sample hyperspectral image, it can be determined whether any pixel in the sample hyperspectral image is in the lesion area to achieve pixel classification and determine the classification result corresponding to each pixel. Using the classification result and spectral features corresponding to each pixel, the feature distance between pixels of the same type can be shortened and the feature distance between pixels of different types can be expanded to achieve comparative learning of spectral information in the pixel dimension, obtain the global semantic relationship between pixels, and determine the contrast loss corresponding to the initial image classification model.

[0071] In one example, the spectral feature corresponding to any pixel is represented as a spectral feature vector, and the contrast loss corresponding to the initial image classification model can be expressed as formula (4):

[0072]

[0073] in, represents the contrast loss corresponding to the initial image classification model; I represents the sample set consisting of all pixels; i represents any pixel in the sample set I; P(i) represents the positive sample set consisting of all pixels whose classification results are the same as pixel i. It should be noted that the positive sample set does not include pixel i; p represents any pixel in the positive sample set P(i); A(i) represents the negative sample set consisting of all pixels whose classification results are different from pixel i; a represents any pixel in the negative sample set; z i represents the spectral feature vector corresponding to pixel i; z prepresents the spectral feature vector corresponding to pixel p; z a represents the spectral feature vector corresponding to pixel a; τ represents a temperature parameter, which is usually set to a constant less than 1 and is used to scale the dot product of the feature vector. Its specific value can be flexibly set according to actual usage requirements, and the present disclosure does not make specific limitations on this.

[0074] In a possible implementation, the initial image classification model is trained according to the contrast loss and the classification loss to determine the trained target image classification model, including: summing the contrast loss and the classification loss according to a preset loss weight to determine the target loss corresponding to the initial image classification model; and training the initial image classification model according to the target loss to determine the target image classification model. The specific value of the loss weight can be flexibly set according to actual usage requirements, and the present disclosure does not make specific limitations on this.

[0075] By using the preset loss weights, the contrast loss and classification loss are weighted and summed. The target loss corresponding to the initial image classification model can be determined by comprehensively considering the pixel dimension and the image dimension, making full use of the spectral information in the sample hyperspectral image. The target loss can be used to train the initial image classification model to obtain a trained target image classification model. Compared with the classification method based on traditional medical images, the target image classification model can be used to distinguish between lesion areas and normal areas in medical hyperspectral images with higher accuracy, sensitivity and specificity.

[0076] Taking the above-mentioned initial image classification model as the U-net neural network model as an example, the target image classification model is obtained through the neural network model training method provided in the embodiment of the present disclosure; as a control, the U-net neural network model is also trained through the neural network model training method based on traditional medical images provided in the prior art to obtain a classification model for traditional medical images. For the same lesion area, medical hyperspectral images and traditional medical images are collected respectively, and the target image classification model and the classification model for traditional medical images are input respectively for image classification, and the corresponding classification results are determined, and the performance is compared from the three dimensions of accuracy, sensitivity and specificity.

[0077] Refer to Table 2, which records, for the same lesion area, the average values ​​and standard deviations of the accuracy, sensitivity and specificity of the classification results when the traditional medical images corresponding to the lesion area are classified using the classification model for traditional medical images, and the average values ​​and standard deviations of the accuracy, sensitivity and specificity of the classification results when the medical hyperspectral images corresponding to the lesion area are classified at the pixel level using the target image classification model.

[0078] Table 2

[0079]

[0080]

[0081] It can be seen that the target image classification model is used to perform pixel-level image classification on the medical hyperspectral image corresponding to the lesion area. The average values ​​of accuracy, sensitivity and specificity corresponding to the classification results are high, and the standard deviation is low, indicating that the performance of the target image classification model is good and relatively stable.

[0082] Figure 3 FIG. 4 shows a box plot of a target image classification model according to an embodiment of the present disclosure. Figure 3 As shown in the figure, the green box plot represents the performance statistics of the classification model for traditional medical images in five dimensions: accuracy, sensitivity, specificity, precision and balanced F score; the blue box plot represents the performance statistics of the target image classification model in five dimensions: accuracy, sensitivity, specificity, precision and balanced F score. Among them, the precision rate (Precision) can describe the classification accuracy for the lesion area; the balanced F score (F1-score), also known as the F1 score, can represent the weighted average of the precision rate and the recall rate (Recall), and the recall rate can describe the proportion of the accurately classified lesion area in all classification results.

[0083] Specifically, the accuracy can be expressed as formula (5):

[0084]

[0085] The recall rate can be expressed as formula (6):

[0086]

[0087] The balanced F score can be expressed as formula (7):

[0088]

[0089] like Figure 3 As shown in the figure, the performance statistics of the target image classification model in five dimensions are better than those of the classification model for traditional medical images, indicating that when the target image classification model is used to perform pixel-level image classification on the medical hyperspectral image corresponding to the lesion area, the classification result has high accuracy and reliability, and can accurately identify the edges of the lesion area and the normal area, providing a reliable image basis for surgical resection, thereby reducing the residual rate of lesion tissue and the retention rate of healthy tissue during surgical resection.

[0090] In one possible implementation, the method further includes: performing a performance evaluation on the target image classification model to determine a performance evaluation result corresponding to the target image classification model; adjusting the loss weight according to the performance evaluation result to determine an adjusted loss weight, wherein the adjusted loss weight is used to retrain the target image classification model.

[0091] In the actual model training process, the preset loss weights may make the performance of the trained target image classification model unable to meet the usage requirements. Therefore, after the training is completed, the target image classification model can be performance evaluated to determine the performance evaluation results corresponding to the target image classification model to determine whether the performance of the target image classification model can meet the usage requirements. Among them, the specific method of performing performance evaluation on the target image classification model can refer to the implementation methods in the relevant technology, and the present disclosure does not make specific limitations on this; the specific form of the performance evaluation results can be flexibly set according to the actual usage requirements, and the present disclosure does not make specific limitations on this.

[0092] In one example, multiple medical hyperspectral images can be input into the target image classification model respectively to determine the classification result corresponding to each medical hyperspectral image, and then combined with the pathology detection result corresponding to each medical hyperspectral image, the classification accuracy of the target image classification model is calculated as the performance evaluation result.

[0093] According to the performance evaluation results, the preset loss weight can be adjusted to obtain the adjusted loss weight, so as to retrain the target image classification model and adjust the performance of the target image classification model. The specific method of adjusting the loss weight can be flexibly set according to the actual use requirements, and this disclosure does not make specific limitations on this.

[0094] In the disclosed embodiment, by inputting the sample hyperspectral image into the initial image classification model, the spectral feature map of the sample hyperspectral image at at least one spatial scale and the initial image classification result corresponding to the sample hyperspectral image can be determined, and then the contrast loss that can reflect the similarity between the spectral features corresponding to any two pixels in the sample hyperspectral image can be determined according to the spectral feature map, so that the contrast learning of global semantic information in the pixel dimension can be realized through the spectral information in the sample hyperspectral image; according to the initial image classification result and the sample label corresponding to the sample hyperspectral image, the classification loss corresponding to the initial image classification model can be determined to provide a classification basis from the image dimension; according to the contrast loss and the classification loss, the initial image classification model is trained to determine the trained target image classification model, so that the target image classification model can make full use of the spectral information in the sample hyperspectral image, perform pixel-level image classification on the lesion area and the normal area in the medical hyperspectral image, improve the accuracy and reliability of the target image classification model, and meet the sensitivity and specificity requirements of medical image classification.

[0095] According to another aspect of the present disclosure, a hyperspectral image classification method is provided.

[0096] Figure 4 A flowchart of a hyperspectral image classification method according to an embodiment of the present disclosure is shown. The hyperspectral image classification method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The hyperspectral image classification method can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the hyperspectral image classification method can be executed by a server. Figure 3 As shown, the hyperspectral image classification method includes:

[0097] In step S41, a medical hyperspectral image corresponding to the target lesion area is acquired.

[0098] The target lesion region may represent any region including lesion tissue that needs to be surgically removed, and its specific form may be flexibly set according to actual use requirements, and the present disclosure does not specifically limit this. Preferably, the target lesion region is a region corresponding to liver tumor tissue.

[0099] Medical hyperspectral images may refer to images acquired by using spectral imaging technology to acquire images of target lesion areas. The specific method of acquiring medical hyperspectral images may refer to the implementation methods in related technologies, and this disclosure does not specifically limit this.

[0100] In step S42, the medical hyperspectral image is classified at the pixel level using the target image classification model to determine the target classification result corresponding to the medical hyperspectral image, wherein the target image classification model is trained by the above-mentioned neural network model training method, and the target classification result is used to indicate the diseased area and the normal area in the medical hyperspectral image.

[0101] By inputting the medical hyperspectral image into the target image classification model, the medical hyperspectral image can be classified at the pixel level to obtain the image that can indicate the diseased area and normal area in the medical hyperspectral image, thereby assisting in identifying the edges of the diseased tissue and healthy tissue in the target lesion area, and providing accurate and reliable image basis for surgical resection of the diseased tissue.

[0102] In the disclosed embodiments, by acquiring a medical hyperspectral image corresponding to a target lesion area and using a target image classification model trained by the above-mentioned neural network model training method, the spectral information in the medical hyperspectral image can be fully utilized to perform pixel-level image classification on the medical hyperspectral image, and determine the target classification result corresponding to the medical hyperspectral image to indicate the lesion area and normal area in the medical hyperspectral image with high accuracy, sensitivity and specificity.

[0103] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0104] In addition, the present disclosure also provides a neural network model training device, a hyperspectral image classification device, an electronic device and a computer-readable storage medium, and a program, all of which can be used to implement any neural network model training method and / or hyperspectral image classification method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0105] Figure 5 A block diagram of a neural network model training device according to an embodiment of the present disclosure is shown. Figure 5 As shown, the device 500 includes:

[0106] The image processing module 501 is used to input the sample hyperspectral image into the initial image classification model, determine the spectral feature map of the sample hyperspectral image at at least one spatial scale, and the initial image classification result corresponding to the sample hyperspectral image;

[0107] A first loss determination module 502 is used to determine the contrast loss corresponding to the initial image classification model according to the spectral feature map, wherein the contrast loss is used to reflect the similarity between the spectral features corresponding to any two pixels in the sample hyperspectral image;

[0108] A second loss determination module 503 is used to determine the classification loss corresponding to the initial image classification model according to the initial image classification result and the sample label corresponding to the sample hyperspectral image;

[0109] The model training module 504 is used to train the initial image classification model according to the contrast loss and the classification loss, and determine the trained target image classification model, wherein the target image classification model is used to perform pixel-level image classification of the lesion area and the normal area in the medical hyperspectral image.

[0110] In one possible implementation, the first loss determination module 502 is specifically used to: unify the spatial scale of the spectral feature map to determine an adjusted spectral feature map, wherein the spatial scale corresponding to the adjusted spectral feature map is the same as the spatial scale corresponding to the sample hyperspectral image; determine the spectral feature corresponding to each pixel in the sample hyperspectral image according to the adjusted spectral feature map; determine the contrast loss according to the spectral feature corresponding to each pixel in the sample hyperspectral image.

[0111] In a possible implementation, the first loss determination module 502 is also used to: classify pixels in the sample hyperspectral image according to the sample label, and determine the classification result corresponding to each pixel, wherein the classification result corresponding to any pixel is used to indicate whether the pixel belongs to the lesion area; determine the contrast loss according to the classification result and spectral characteristics corresponding to each pixel.

[0112] In one possible implementation, the model training module 504 is specifically used to: sum the contrast loss and the classification loss according to a preset loss weight to determine the target loss corresponding to the initial image classification model; and train the initial image classification model according to the target loss to determine the target image classification model.

[0113] In one possible implementation, the device 500 also includes a loss weight adjustment module, which is used to: perform a performance evaluation on the target image classification model to determine a performance evaluation result corresponding to the target image classification model; adjust the loss weight according to the performance evaluation result to determine the adjusted loss weight, wherein the adjusted loss weight is used to retrain the target image classification model.

[0114] Figure 6 FIG. 5 is a block diagram of a hyperspectral image classification device according to an embodiment of the present disclosure. Figure 6 As shown, the device 600 includes:

[0115] An image acquisition module 601 is used to acquire a medical hyperspectral image corresponding to a target lesion area;

[0116] The image classification module 602 is used to perform pixel-level image classification on the medical hyperspectral image using a target image classification model to determine a target classification result corresponding to the medical hyperspectral image, wherein the target image classification model is trained by the above-mentioned neural network model training method, and the target classification result is used to indicate the diseased area and the normal area in the medical hyperspectral image.

[0117] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0118] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0119] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0120] Figure 7 1900 is a block diagram of an electronic device according to an embodiment of the present disclosure. For example, the apparatus 1900 may be provided as a server or a terminal device. Figure 7 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0121] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM, FreeBSD TM or similar.

[0122] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0123] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0124] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0125] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0126] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0127] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0128] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0129] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0130] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0131] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A neural network model training method, characterized in that: include: Inputting the sample hyperspectral image into the initial image classification model, determining a spectral feature map of the sample hyperspectral image at at least one spatial scale, and an initial image classification result corresponding to the sample hyperspectral image; Determining the contrast loss corresponding to the initial image classification model according to the spectral feature map, wherein the contrast loss is used to reflect the similarity between the spectral features corresponding to any two pixels in the sample hyperspectral image; Determining a classification loss corresponding to the initial image classification model according to the initial image classification result and a sample label corresponding to the sample hyperspectral image; The initial image classification model is trained according to the contrast loss and the classification loss to determine a trained target image classification model, wherein the target image classification model is used to perform pixel-level image classification of lesion areas and normal areas in medical hyperspectral images.

2. The method according to claim 1, characterized in that Determining the contrast loss corresponding to the initial image classification model according to the spectral feature map includes: Unifying the spatial scale of the spectral characteristic graph to determine an adjusted spectral characteristic graph, wherein the spatial scale corresponding to the adjusted spectral characteristic graph is the same as the spatial scale corresponding to the sample hyperspectral image; Determining the spectral feature corresponding to each pixel in the sample hyperspectral image according to the adjusted spectral feature map; The contrast loss is determined according to the spectral feature corresponding to each pixel in the sample hyperspectral image.

3. The method according to claim 2, characterized in that The step of determining the contrast loss according to the spectral feature corresponding to each pixel in the sample hyperspectral image includes: Classifying pixels in the sample hyperspectral image according to the sample label to determine a classification result corresponding to each pixel, wherein the classification result corresponding to any pixel is used to indicate whether the pixel belongs to the lesion area; The contrast loss is determined according to the classification result and the spectral feature corresponding to each pixel.

4. The method according to any one of claims 1 to 3, characterized in that The step of training the initial image classification model according to the contrast loss and the classification loss to determine a trained target image classification model includes: According to a preset loss weight, the contrast loss and the classification loss are summed to determine a target loss corresponding to the initial image classification model; The initial image classification model is trained according to the target loss to determine the target image classification model.

5. The method according to claim 4, characterized in that The method further comprises: Performing a performance evaluation on the target image classification model to determine a performance evaluation result corresponding to the target image classification model; According to the performance evaluation result, the loss weight is adjusted to determine the adjusted loss weight, wherein the adjusted loss weight is used to retrain the target image classification model.

6. A hyperspectral image classification method, characterized in that: include: Acquire a medical hyperspectral image corresponding to the target lesion area; Using a target image classification model, the medical hyperspectral image is subjected to pixel-level image classification to determine a target classification result corresponding to the medical hyperspectral image, wherein the target image classification model is trained by the method described in any one of claims 1 to 5, and the target classification result is used to indicate a diseased area and a normal area in the medical hyperspectral image.

7. A neural network model training device, characterized in that: include: An image processing module, used for inputting a sample hyperspectral image into an initial image classification model, determining a spectral feature map of the sample hyperspectral image at at least one spatial scale, and an initial image classification result corresponding to the sample hyperspectral image; A first loss determination module, used to determine the contrast loss corresponding to the initial image classification model according to the spectral feature map, wherein the contrast loss is used to reflect the similarity between the spectral features corresponding to any two pixels in the sample hyperspectral image; A second loss determination module is used to determine the classification loss corresponding to the initial image classification model according to the initial image classification result and the sample label corresponding to the sample hyperspectral image; The model training module is used to train the initial image classification model according to the contrast loss and the classification loss, and determine a trained target image classification model, wherein the target image classification model is used to perform pixel-level image classification of lesion areas and normal areas in medical hyperspectral images.

8. A hyperspectral image classification device, characterized in that: include: An image acquisition module, used to acquire a medical hyperspectral image corresponding to a target lesion area; An image classification module is used to perform pixel-level image classification on the medical hyperspectral image using a target image classification model to determine a target classification result corresponding to the medical hyperspectral image, wherein the target image classification model is trained by the method described in any one of claims 1 to 5, and the target classification result is used to indicate a diseased area and a normal area in the medical hyperspectral image.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 6 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images