Medical data classification method and device, equipment, storage medium and product

Abstract

The invention relates to the technical field of computers, and discloses a medical data classification method and device, equipment, a storage medium and a product. The method comprises the following steps: inputting multi-modal medical data of a target object into a trained medical classification model; wherein the multi-modal medical data comprises at least two kinds of modal data in image feature data, full-slice feature data, index parameter data and gene sequencing data, and the medical classification model comprises a classification output network and feature extraction networks corresponding to the at least two kinds of modal data; for each type of modal data, inputting the modal data into a corresponding feature extraction network to obtain an output modal feature vector; and inputting the at least two modal feature vectors into a classification output network to obtain a classification label corresponding to the output multi-modal medical data, thereby solving the problem that a traditional deep learning model is insufficient in adaptability to a complex classification scene, and improving the classification accuracy of the medical data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a medical data classification method, device, equipment, storage medium and product. Background Art

[0002] At present, deep learning models have been widely used in the medical field with their powerful pattern recognition and data processing capabilities, aiming to provide strong data support and technical guarantees for medical research and medical information management.

[0003] Due to the diversity of clinical information presented by different individuals, single-modality medical data makes the existing deep learning models insufficiently adaptable to complex classification scenarios and difficult to meet the high requirements of the medical field for classification accuracy. Excessive classification errors lead to poor operating efficiency and stability of medical information systems. Summary of the invention

[0004] The embodiments of the present invention provide a medical data classification method, apparatus, equipment, storage medium and product to solve the problem that traditional deep learning models are insufficiently adaptable to complex classification scenarios, improve the classification accuracy of medical data, and thereby ensure the operating efficiency and stability of the medical information system.

[0005] According to an embodiment of the present invention, a method for classifying medical data is provided, the method comprising:

[0006] Inputting the multimodal medical data of the target object into a trained medical classification model; wherein the multimodal medical data includes at least two modal data of image feature data, full-slice feature data, indicator parameter data, and gene sequencing data, and the medical classification model includes a classification output network and a feature extraction network corresponding to the at least two modal data respectively;

[0007] For each type of modal data, the modal data is input into a corresponding feature extraction network to obtain an output modal feature vector;

[0008] At least two modality feature vectors are input into the classification output network to obtain classification labels corresponding to the output multimodal medical data.

[0009] According to another embodiment of the present invention, a medical data classification device is provided, the device comprising:

[0010] A multimodal medical data input module, used to input the multimodal medical data of the target object into a trained medical classification model; wherein the multimodal medical data includes at least two modal data of image feature data, full-slice feature data, indicator parameter data and gene sequencing data, and the medical classification model includes a classification output network and a feature extraction network corresponding to the at least two modal data respectively;

[0011] A modal feature vector output module, for each type of modal data, inputs the modal data into a corresponding feature extraction network to obtain an output modal feature vector;

[0012] The classification label output module is used to input at least two modality feature vectors into the classification output network to obtain the classification labels corresponding to the output multimodal medical data.

[0013] According to another embodiment of the present invention, there is provided an electronic device, the electronic device comprising:

[0014] at least one processor; and a memory communicatively coupled to the at least one processor;

[0015] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the medical data classification method described in any embodiment of the present invention.

[0016] According to another embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the medical data classification method described in any embodiment of the present invention when executed.

[0017] According to another embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for classifying medical data described in any embodiment of the present invention is implemented.

[0018] The technical solution of the embodiment of the present invention is to input at least two modal data of the image feature data, full-slice feature data, indicator parameter data and gene sequencing data corresponding to the target object into a trained medical classification model, and for each modal data, input the modal data into a feature extraction network corresponding to the modal data in the medical classification model to obtain an output modal feature vector, and input at least two modal feature vectors into a classification output network in the medical classification model to obtain a classification label corresponding to the output multimodal medical data, thereby realizing information complementarity between medical data of different modalities, enriching the data feature dimension, solving the problem of insufficient adaptability of traditional deep learning models to complex classification scenarios, improving the classification accuracy of medical data, and thereby ensuring the operating efficiency and stability of the medical information system.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flowchart of a method for classifying medical data provided by one embodiment of the present invention;

[0022] Figure 2 A schematic diagram of the architecture of a medical classification model provided by one embodiment of the present invention;

[0023] Figure 3 A flowchart of another method for classifying medical data provided by one embodiment of the present invention;

[0024] Figure 4 A flowchart of a specific example of a method for acquiring multimodal medical data provided by an embodiment of the present invention;

[0025] Figure 5 A flowchart of another method for classifying medical data provided by one embodiment of the present invention;

[0026] Figure 6 A schematic diagram of the architecture of a graph feature coding layer provided by an embodiment of the present invention;

[0027] Figure 7A schematic diagram of the structure of a medical data classification device provided by an embodiment of the present invention;

[0028] Figure 8 The present invention is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "initial", "level", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Figure 1 This is a flowchart of a medical data classification method provided by an embodiment of the present invention. This embodiment is applicable to the case of classifying medical data. The method can be executed by a medical data classification device. The medical data classification device can be implemented in the form of hardware and / or software. The medical data classification device can be configured in a terminal device. Figure 1 As shown, the method includes:

[0032] S110, inputting the multimodal medical data of the target object into the trained medical classification model.

[0033] In this embodiment, the multimodal medical data includes at least two modality data of image feature data, whole slice feature data, indicator parameter data and gene sequencing data.

[0034] Specifically, the image feature data represents feature information corresponding to a tomographic image collected by a tomographic medical imaging device. For example, the medical imaging device may be a direct digital radiography system (DR), a computed tomography device (CT), a magnetic resonance imaging device (MRI), a positron emission computed tomography device (PET), or an ultrasound device, etc., but is not limited to the example scenarios.

[0035] In an optional embodiment, the image feature data includes scanned image features and / or radiomics features corresponding to a tomographic image of the target object, wherein the tomographic image is read from a storage system, the scanned image feature representation is a feature that can express semantic information mined and extracted from the tomographic image using a specific algorithm or network model, and the radiomics feature representation is a quantitative description of the inherent attributes of the tomographic image.

[0036] Specifically, an omics analysis tool is used to extract the imaging omics features of the tomographic image. Exemplarily, the omics analysis tool may be a Pyradiomics library, a SimpleITK library, or a Scikit Image library, but is not limited to the example case.

[0037] In an optional embodiment, the method further includes: performing image segmentation training on the encoder-decoder model based on the tomographic image; inputting the tomographic image into the encoder in the trained encoder-decoder model to obtain scanning image features corresponding to the tomographic image output by the encoder.

[0038] Specifically, the tomographic image is the input data of the encoder-decoder model, and the training label data is the regional image or regional boundary data corresponding to the region of interest in the tomographic image. Exemplarily, the encoder-decoder model can be a U-Net model or its deformation, a recurrent neural network or its deformation, etc., but is not limited to the example case.

[0039] In a specific embodiment, when the encoder-decoder model is a U-Net model or a deformation thereof, the downsampling operation in the encoder-decoder model is implemented through a residual structure to improve the feature extraction capability of the encoder-decoder model.

[0040] In a specific embodiment, the encoder-decoder model utilizes a deep supervised training strategy to improve the processing capability of the encoder-decoder model at different levels, and / or uses a transfer learning method to improve the model performance of the encoder-decoder model and enhance robustness.

[0041] Specifically, whole-slide feature data characterizes the feature information corresponding to whole-slide images (WSI). Whole-slide images are the product of digitizing traditional glass histology slices, aiming to fully and accurately present the slice information of the entire tissue. They usually have extremely high resolution and a large number of pixels. Generally speaking, the pixels of a whole-slide image may reach billions or even higher.

[0042] In an optional embodiment, the full-slice feature data includes a slice graph structure and / or slice omics features corresponding to the full-slice image of the target object. The full-slice image is read from a storage system, the slice graph structure represents a graph structure constructed based on image block features of multiple slice image blocks corresponding to the full-slice image, and the slice omics features represent a quantitative description of the inherent properties of the full-slice image. The image block feature representation uses a specific algorithm or network model to mine and extract features that can express semantic information from slice image blocks.

[0043] Specifically, an omics analysis tool is used to extract slice omics features corresponding to the slice image block set. Exemplarily, the omics analysis tool may be a Pyradiomics library, a SimpleITK library, or a Scikit Image library, but is not limited to the example case.

[0044] Specifically, the indicator parameter data and gene sequencing data are read from the storage system. Among them, the indicator parameter data represents the indicator information used to reflect the physical condition of the target object. Exemplarily, the indicator parameter data includes but is not limited to age, blood pressure, blood sugar and body temperature, etc. The gene sequencing data represents the base sequence information obtained by genetic testing and analysis of the target object. Exemplarily, the genome sequencing data includes but is not limited to transcriptome sequencing data, whole genome sequencing data and exome sequencing data, etc.

[0045] There are no restrictions on indicator parameter data and genome sequencing data here, and specific settings can be customized according to actual needs.

[0046] In this embodiment, the medical classification model includes a classification output network and a feature extraction network corresponding to at least two modal data. Specifically, the network structures of the feature extraction networks corresponding to different modal data can be the same or different.

[0047] S120. For each type of modal data, the modal data is input into a corresponding feature extraction network to obtain an output modal feature vector.

[0048] In an optional embodiment, when the image feature data includes scanned image features or radiomics features, the feature extraction network corresponding to the image feature data is an attention network.

[0049] In another optional embodiment, when the image feature data includes scanned image features and imaging genomics features, the feature extraction network corresponding to the image feature data includes a bilinear layer and an attention layer. Accordingly, the modal data is input into the corresponding feature extraction network to obtain an output modal feature vector, including: inputting the scanned image features and the imaging genomics features into the bilinear layer to obtain output image fusion features; inputting the image fusion features into the attention layer for attention processing to obtain the modal feature vector corresponding to the output image feature data.

[0050] Exemplary image fusion features Satisfies the following formula:

[0051] ;

[0052] in, Represents the scanned image features, Represents the radiomics features, represents a bilinear layer, Represents the learnable weight matrix of the bilinear layer.

[0053] The advantage of this setting is that it can achieve intra-group feature fusion of imaging modalities and ensure the classification accuracy of medical data.

[0054] In an optional embodiment, when the full-slice feature data includes a slice graph structure or a slice omics feature, the feature extraction network corresponding to the full-slice feature data is an attention network.

[0055] In an optional embodiment, the feature extraction network corresponding to the indicator parameter data and the feature extraction network corresponding to the gene sequencing data are attention networks.

[0056] In an optional embodiment, the attention network or attention layer adopts a self-attention mechanism. Exemplarily, the self-attention mechanism is expressed as:

[0057] ;

[0058] ;

[0059] ;

[0060] ;

[0061] in, represents the input sequence, is a learnable weight matrix that transforms the input sequence Projected into a unified feature space, , Represents the query vector The dimension of , Represents the key vector The dimension of , Represents a value vector The dimension of Represents the input sequence The vector dimension of . represents the attention score, Represents the scaling factor, which is used to prevent the gradient from disappearing or exploding and ensure the stability of softmax. represents the attention weight, Represents the output matrix of the self-attention mechanism.

[0062] S130. Input at least two modal feature vectors into a classification output network to obtain classification labels corresponding to the output multimodal medical data.

[0063] In an optional embodiment, the classification output network includes a concatenation layer and a linear layer, wherein the concatenation layer is used to concatenate at least two modality feature vectors to obtain concatenated modality features, and the linear layer is used to determine the classification label corresponding to the multimodal medical data according to the concatenated modality features.

[0064] In an alternative embodiment, the classification label represents the likelihood that a given drug will produce evidence of effective action on a target subject.

[0065] Different individuals are affected by factors such as genetic factors, lifestyle habits, and past medical history. The likelihood of a given drug producing effective signs on different individuals in the same specific environment varies. Therefore, single and simple medical data is not sufficient to fully explain the heterogeneity of a given drug in different individuals in the same specific environment. In this embodiment, the classification label can be used to guide drug research on a given drug to determine the effectiveness and safety of the given drug.

[0066] Figure 2The schematic diagram of the architecture of a medical classification model provided by an embodiment of the present invention. Specifically, the medical classification model includes feature extraction networks corresponding to the index parameter data, gene sequencing data, image feature data, and full-slice feature data, wherein the feature extraction networks corresponding to the index parameter data and gene sequencing data are attention networks, and when the image feature data include scanned image features and radiomics features, the feature extraction network corresponding to the image feature data in the medical classification model includes a bilinear layer and an attention layer. The classification output network in the medical classification model ( Figure 2 The dashed box in the figure contains the concatenation layer and the linear layer.

[0067] Based on the above embodiments, optionally, the method also includes: inputting the training medical data set into an untrained medical classification model to obtain an output prediction label set; determining a loss function value based on the prediction label set and the standard label set, and iteratively adjusting the model parameters of the medical classification model based on the loss function value; until the loss function value converges, the medical classification model in the current iteration process is used as the trained medical classification model.

[0068] The training medical data set contains multiple multimodal medical data for training, the predicted label set contains the predicted classification label corresponding to each multimodal medical data, and the standard label set contains the true classification label corresponding to each multimodal medical data.

[0069] Exemplarily, the loss function corresponding to the loss function value can be a square loss function, a logarithmic loss function, an exponential loss function, a mean square error loss function, a logistic regression loss function, a Huber loss function, a cross entropy loss function, and a Kullback-Leibler divergence loss function, etc., but is not limited to the example scenario.

[0070] The technical solution of this embodiment is to input at least two modal data of the image feature data, full-slice feature data, indicator parameter data and gene sequencing data corresponding to the target object into a trained medical classification model, and for each modal data, input the modal data into a feature extraction network corresponding to the modal data in the medical classification model to obtain an output modal feature vector, and input at least two modal feature vectors into a classification output network in the medical classification model to obtain a classification label corresponding to the output multimodal medical data, thereby realizing information complementarity between medical data of different modalities, enriching the data feature dimension, solving the problem of insufficient adaptability of traditional deep learning models to complex classification scenarios, improving the classification accuracy of medical data, and thereby ensuring the operating efficiency and stability of the medical information system.

[0071] Figure 3This is a flow chart of another medical data classification method provided by an embodiment of the present invention. This embodiment further refines the method for obtaining the slice graph structure in the above embodiment. Figure 3 As shown, the method includes:

[0072] S210 , segmenting the full slice image of the target object to obtain a slice image block set.

[0073] Specifically, the slice image block set includes a plurality of slice image blocks. Exemplarily, the size of the slice image block may be 256×256 or 512×512. The size of the slice image block is not limited here and may be customized according to actual needs.

[0074] S220 , training an abnormal tissue recognition model according to the slice image block set, and determining an image block feature set according to the abnormal tissue recognition model obtained by the training.

[0075] In an optional embodiment, before training the abnormal tissue recognition model based on the slice image block set, the method also includes: for each slice image block, binarizing the slice image block to obtain a binary image block; determining the pixel ratio corresponding to the background pixel points in the binary image block; and when the pixel ratio is greater than a ratio threshold, deleting the slice image block from the slice image block.

[0076] Exemplarily, the binarization algorithm includes but is not limited to the Otsu algorithm, the maximum entropy threshold method, the Niblack algorithm, the Sauvola algorithm or the Bernsen algorithm, etc. The ratio threshold may be 0.75 or 0.8, but is not limited to the example scenario.

[0077] Specifically, the background pixel is a pixel with a pixel value of 1 in the binary image block, and can also be a pixel with a pixel value of 0 in the binary image block. The pixel ratio represents the ratio of the number of background pixels to the total number of pixels in the binary image block.

[0078] The advantage of this setting is that, on the one hand, it reduces the noise interference caused by a large amount of background information on the training of the abnormal tissue recognition model, thereby ensuring the training effect of the abnormal tissue recognition model; on the other hand, it reduces the data volume of the slice image block set, thereby improving the training efficiency of the abnormal tissue recognition model.

[0079] Specifically, the slice image block set is the input data of the abnormal tissue recognition model, and the training label data is the classification label of the slice image block, and the classification label represents whether the slice image block contains abnormal tissue. Exemplarily, the abnormal tissue recognition model can be a residual network model, a deep separable convolutional model or an EfficientNet model, but is not limited to the example case.

[0080] In a specific embodiment, when the abnormal tissue recognition model is an EfficientNet model, the output result of the last convolutional layer in the EfficientNet model is used as an image block feature set, wherein the image block feature set includes image block features corresponding to each slice image block in the slice image block set.

[0081] S230 , performing similarity screening on the image block feature set to obtain a screened image block feature set, and determining a slice graph structure according to the screened image block feature set.

[0082] In an optional embodiment, similarity screening is performed on an image block feature set to obtain a screened image block feature set, including: determining at least one slice image region according to the slice image block set; wherein the slice image region includes a central slice image block and at least two adjacent slice image blocks corresponding to the first slice image block; for each slice image region, obtaining a first image block feature corresponding to the central slice image block in the slice image region and a second image block feature corresponding to the at least two adjacent slice image blocks in the image feature set; determining a feature similarity corresponding to each adjacent slice image block and the central slice image block according to the first image block feature and the at least two second image block features; and screening the image block feature set according to the at least two feature similarities corresponding to each slice image region to obtain a screened image block feature set.

[0083] Specifically, the adjacent slice image blocks represent slice image blocks whose image block distance from the central slice image block is less than a distance threshold. Exemplarily, the slice image region may be a 5×5 image region, that is, the slice image region includes at most 25 slice image blocks.

[0084] Specifically, the second image block features correspond to the feature similarity one by one. Exemplarily, the similarity algorithm may be Euclidean distance, Manhattan distance, or cosine similarity, but is not limited to the example case. When the similarity algorithm is Euclidean distance, the feature similarity Satisfies the following formula:

[0085] ;

[0086] When the similarity algorithm is cosine similarity, feature similarity Satisfies the following formula:

[0087] ;

[0088] in, Indicates the feature dimension corresponding to the image block feature, Represents the first image block feature The A vector represents Represents the second image block feature The Vector representation.

[0089] In an optional embodiment, the image block feature set is screened according to at least two feature similarities corresponding to each slice image region to obtain a screened image block feature set, including: determining the number of similarities corresponding to each slice image region according to a similarity threshold and at least two feature similarities corresponding to each slice image region; sorting the at least two slice image regions in descending order according to the number of similarities to obtain sorted region data; and screening the image block feature set according to the sorted region data to obtain a screened image block feature set.

[0090] The similarity number indicates the number of adjacent slice image blocks in the slice image region whose feature similarity exceeds the similarity threshold. For example, the similarity threshold may be 0.8 or 0.9, but is not limited to the example case.

[0091] The advantage of this setting is that it improves the screening efficiency of the image block feature set, thereby improving the processing efficiency of the full slice image.

[0092] In this embodiment, the nodes in the slice graph structure represent image block features, and the edges between the nodes represent that the distance between two slice image blocks is less than a distance threshold.

[0093] S240, inputting the multimodal medical data including the slice graph structure into the trained medical classification model.

[0094] Figure 4 A flowchart of a specific example of a method for acquiring multimodal medical data provided by an embodiment of the present invention. Specifically, the full slice image of the target object is segmented to obtain a slice image block set, and the slice omics features corresponding to the slice image block set are extracted using the Pyradiomics library. And according to the slice image block set, the abnormal tissue recognition model is trained, and the image block feature set is determined according to the abnormal tissue recognition model obtained by training, and the image block feature set is screened for similarity to obtain the screened image block feature set, and the slice image structure is determined according to the screened image block feature set, and the full slice feature data is determined according to the slice omics features and the slice image structure.

[0095] The Pyradiomics library is used to extract the radiomic features of the tomographic image of the target object. The encoder-decoder model is trained for image segmentation based on the tomographic image, and the tomographic image is input into the encoder of the trained encoder-decoder model to obtain the scanned image features corresponding to the tomographic image output by the encoder. The image feature data is determined based on the radiomic features and the scanned image features.

[0096] The image feature data, whole-slice feature data, indicator parameter data and gene sequencing data of the target object are input into the medical classification model.

[0097] S250: For each type of modal data, the modal data is input into a corresponding feature extraction network to obtain an output modal feature vector.

[0098] S260: Input at least two modal feature vectors into a classification output network to obtain classification labels corresponding to the output multimodal medical data.

[0099] S250-S260 in this embodiment are similar to those in the above embodiment. Figure 1 The S120 - S130 shown correspond to the same or similar ones, and are not described in detail in this embodiment.

[0100] The number of pixels in a full slice image may reach billions or even higher, and the slice image blocks can only express tiny local features. Currently, deep learning techniques for full slice images mostly use multi-instance learning methods, which leads to the neglect of information interaction between different slice image blocks.

[0101] This embodiment generates a slice graph structure based on the full slice image and uses the slice graph structure as input data of the medical classification model, which not only reduces the data volume of multimodal medical data but also retains the global key information of the full slice image as much as possible, thereby improving the classification efficiency and classification accuracy of the medical data at the same time.

[0102] Figure 5 This is a flow chart of another medical data classification method provided by an embodiment of the present invention. This embodiment further refines the "feature extraction network corresponding to the full slice feature data" in the above embodiment. In this embodiment, when the full slice feature data includes a slice graph structure and slice omics features, the feature extraction network corresponding to the full slice feature data includes a graph feature encoding layer, a bilinear layer, and an attention layer; the modal data is input into the corresponding feature extraction network to obtain an output modal feature vector, including: inputting the slice graph structure into the graph feature encoding layer for feature encoding to obtain output slice coding features; inputting slice coding features and slice omics features into the bilinear layer for linear fusion to obtain output slice fusion features; inputting slice fusion features into the attention layer for attention processing to obtain an output modal feature vector corresponding to the full slice feature data. Figure 5 As shown, the method includes:

[0103] S310: Input the multimodal medical data of the target object into the trained medical classification model.

[0104] In this embodiment, the multimodal medical data at least includes full-slice feature data, and the full-slice feature data includes a slice graph structure and slice omics features.

[0105] S310 in this embodiment is similar to that in the above embodiment. Figure 1 The S110 shown corresponds to the same or similar steps, and will not be described in detail in this embodiment.

[0106] S320, inputting the modal data except the full-slice feature data into the corresponding feature extraction network to obtain an output modal feature vector.

[0107] S320 in this embodiment is similar to that in the above embodiment. Figure 1 The S120 shown corresponds to the same or similar steps, and will not be described in detail in this embodiment.

[0108] S330: Input the slice graph structure in the full slice feature data into the graph feature coding layer for feature coding to obtain output slice coding features.

[0109] In this embodiment, the feature extraction network corresponding to the full-slice feature data includes a graph feature encoding layer, a bilinear layer, and an attention layer.

[0110] In an optional embodiment, the graph feature encoding layer is a graph convolutional network (GCN), a graph attention network (GAT), a graph recurrent network (GRN), or a gated graph neural network (GGNN), but is not limited to the example scenario.

[0111] In another optional embodiment, the graph feature encoding layer includes a multi-layer perceptron, at least one graph feature extraction layer, a splicing layer and a graph structure encoding layer; the slice graph structure is input into the graph feature encoding layer for feature encoding to obtain output slice encoding features, including: inputting the slice graph structure into the multi-layer perceptron layer to obtain output initial graph structure features; extracting features from the initial graph structure features through at least one graph feature extraction layer connected in series to obtain at least one hierarchical graph structure feature; inputting the initial graph structure features and at least one hierarchical graph structure feature into the splicing layer to obtain output spliced ​​graph structure features; inputting the spliced ​​graph structure features into the graph structure encoding layer for feature encoding to obtain output slice encoding features.

[0112] Among them, the Multilayer Perceptron (MLP) is a feedforward artificial neural network model, which consists of at least three layers of nodes, namely an input layer, at least one hidden layer and an output layer.

[0113] Among them, at least one graph feature extraction layer in the graph feature encoding layer is connected in series in sequence, the input data of the first graph feature extraction layer is the initial graph structure feature, the input data of the graph feature extraction layers other than the first graph feature extraction layer is the hierarchical graph structure feature output by the previous image feature extraction layer, and the graph feature extraction layers correspond to the hierarchical graph structure features one by one.

[0114] In an optional embodiment, the graph feature extraction layer is a graph convolutional network, a graph attention network, a graph recurrent network, or a gated graph neural network, but is not limited to the example scenario.

[0115] In another optional embodiment, the graph feature extraction layer includes a graph attention module, a graph convolution module and a fusion module; wherein the graph attention module is used to perform graph attention processing on the input data to obtain output graph attention features, the graph convolution module is used to perform graph convolution processing on the input data to obtain output graph convolution features, and the fusion module is used to perform feature fusion on the graph attention features and the graph convolution features to obtain output hierarchical graph structure features.

[0116] In an optional embodiment, the graph attention module includes a graph attention convolution layer (GraphAttention Convolution layer, GATConv), a layer normalization (Layer Norm), and a parametric rectified linear unit (PReLU) connected in series.

[0117] For example, the graph attention mechanism can be expressed as:

[0118] ;

[0119] ;

[0120] ;

[0121] in, and Represents a node in a graph structure and its neighbor nodes The feature vector of , Represents a learnable weight matrix, which is used to transform and Mapped to the new feature space, we get and .in, , Represents the output feature dimension. represents the attention coefficient, represents the attention weight and the node For Node The importance of Is a node The set of neighbor nodes (including itself), is a learnable weight vector, , represents the vector concatenation operation, represents a nonlinear activation function, Representation Node Updated feature vector.

[0122] In an optional embodiment, the graph convolution module includes at least two graph convolution units with residual connections, wherein the input data of the first graph convolution unit is the input data of the graph convolution module, the input data of the graph convolution units other than the first graph convolution unit are the input data and output results of the previous graph convolution unit, and the output result of the last graph convolution unit is the graph convolution feature.

[0123] In an optional embodiment, the graph convolution unit includes a generalized graph convolution layer (Generalized Graph Convolution, GENConv), a layer normalization and a rectified linear unit (Rectified Linear Unit, ReLU) connected in series.

[0124] Exemplarily, the graph convolution mechanism can be expressed as:

[0125] ;

[0126] ;

[0127] ;

[0128] in, Represents the central node , neighbor nodes and the corresponding edge The message passing function between Represents the central node In the Layer (or Iterations), Neighbor node In the Layer (or iterations), Represents the central node , neighbor nodes Between In the Layer (or The feature vector of the central node , neighbor nodes There is no edge between them. The value of is 0. Represents the central node , neighbor nodes and the corresponding edge The feature information between is an error constant with a value of 10 -7 . It is calculated Aggregate information The contribution value of represents the learnable parameters, Represents the final aggregation operation, Represents the central node The set of neighbor nodes (including itself), represents a multi-layer perceptron, Represents the central node In the Layer (or Iterations).

[0129] In an optional embodiment, the splicing layer is used to perform global average pooling processing on the initial graph structure feature and at least one hierarchical graph structure feature respectively, and to splice at least two processing results to obtain a spliced ​​graph structure feature.

[0130] In an optional embodiment, the graph structure encoding layer is composed of a multi-layer perceptron, a multi-head self-attention module and a linear layer connected in series.

[0131] Figure 6 A schematic diagram of the architecture of a graph feature coding layer provided for an embodiment of the present invention. Specifically, the graph feature coding layer includes a multi-layer perceptron, 5 graph feature extraction layers, 1 splicing layer and a graph structure coding layer. Among them, the graph feature extraction layer includes a graph attention module, a graph convolution module and a fusion module. The graph attention module is composed of GATConv, layer normalization and PReLU. The graph convolution module is composed of two graph convolution units with residual connections. The graph convolution unit is composed of GENConv, layer normalization and ReLU. The fusion module is a residual module.

[0132] like Figure 6As shown in the figure, in the splicing layer, the initial graph structure features output by the multi-layer perceptron are processed by global average pooling to obtain scale1, and the hierarchical graph structure features output by each graph feature extraction layer are processed by global average pooling to obtain scale2-scale6. Scale1-scale6 are spliced ​​to obtain the spliced ​​graph structure features. The spliced ​​graph structure features are input into the graph structure encoding layer, and feature encoding is performed by the multi-layer perceptron, multi-head self-attention module and linear layer in sequence to obtain the output slice encoding features.

[0133] S340, inputting the slice coding features and the slice omics features in the full slice feature data into a bilinear layer for linear fusion to obtain output slice fusion features.

[0134] Exemplary, slice fusion features Satisfies the following formula:

[0135] ;

[0136] in, represents the slice encoding feature, represents the slice omics features, represents a bilinear module, Represents the learnable weight matrix of the bilinear module.

[0137] S350: Input the slice fusion features into the attention layer for attention processing to obtain the modal feature vector corresponding to the output full slice feature data.

[0138] In an optional embodiment, the attention module adopts a self-attention mechanism.

[0139] S360: Input at least two modal feature vectors into a classification output network to obtain classification labels corresponding to the output multimodal medical data.

[0140] S360 in this embodiment is the same as that in the above embodiment. Figure 1 The S130 shown corresponds to the same or similar steps, and will not be described in detail in this embodiment.

[0141] The technical solution of this embodiment is to input the slice graph structure into the graph feature coding layer for feature coding to obtain output slice coding features, input the slice coding features and the slice omics features into the bilinear layer for linear fusion to obtain output slice fusion features, input the slice fusion features into the attention layer for attention processing to obtain the output modal feature vector corresponding to the full slice feature data, and the bilinear feature module realizes the fusion purpose of the intra-group modality of the full slice feature data, extracts deep feature information of the slice graph structure, and further improves the classification accuracy of medical data.

[0142] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0143] The following is an embodiment of a medical data classification device provided in an embodiment of the present invention. The device and the medical data classification method of the above-mentioned embodiment belong to the same inventive concept. For details not described in detail in the embodiment of the medical data classification device, reference can be made to the contents of the medical data classification method in the above-mentioned embodiment.

[0144] Figure 7 A schematic diagram of the structure of a medical data classification device provided by an embodiment of the present invention. Figure 7 As shown, the device includes: a multimodal medical data input module 410, a modality feature vector output module 420 and a classification label output module 430.

[0145] The multimodal medical data input module 410 is used to input the multimodal medical data of the target object into the trained medical classification model; wherein the multimodal medical data includes at least two modal data of image feature data, full slice feature data, index parameter data and gene sequencing data, and the medical classification model includes a classification output network and a feature extraction network corresponding to the at least two modal data respectively;

[0146] A modal feature vector output module 420 is used to input the modal data into a corresponding feature extraction network for each modal data to obtain an output modal feature vector;

[0147] The classification label output module 430 is used to input at least two modality feature vectors into the classification output network to obtain classification labels corresponding to the output multimodal medical data.

[0148] The technical solution of this embodiment realizes information complementarity between medical data of different modalities, enriches the data feature dimensions, solves the problem of insufficient adaptability of traditional deep learning models to complex classification scenarios, improves the classification accuracy of medical data, and thus ensures the operating efficiency and stability of the medical information system.

[0149] In an optional embodiment, the full-slice feature data includes a slice graph structure and / or slice omics features corresponding to the full-slice image of the target object, and accordingly, the device further includes:

[0150] A slice image structure determination module is used to segment the full slice image of the target object to obtain a slice image block set;

[0151] The abnormal tissue recognition model is trained according to the slice image block set, and the image block feature set is determined according to the abnormal tissue recognition model obtained by the training; wherein the image block feature set includes image block features corresponding to each slice image block in the slice image block set;

[0152] Performing similarity screening on the image block feature set to obtain a screened image block feature set, and determining a slice graph structure based on the screened image block feature set;

[0153] The nodes in the slice graph structure represent image block features, and the edges between the nodes represent that the distance between two slice image blocks is less than a distance threshold.

[0154] In an optional embodiment, when the full-slice feature data includes a slice graph structure and a slice omics feature, the feature extraction network corresponding to the full-slice feature data includes a graph feature encoding layer, a bilinear layer, and an attention layer; the modal feature vector output module 420 includes:

[0155] A modal feature vector determination unit is used to input the slice graph structure into the graph feature encoding layer for feature encoding, and obtain an output slice encoding feature;

[0156] The slice coding features and slice omics features are input into the bilinear layer for linear fusion to obtain the output slice fusion features;

[0157] The slice fusion features are input into the attention layer for attention processing to obtain the modal feature vector corresponding to the output full slice feature data.

[0158] In an optional embodiment, the graph feature encoding layer includes a multi-layer perceptron, at least one graph feature extraction layer, a concatenation layer and a graph structure encoding layer; the modal feature vector determination unit is specifically used to:

[0159] Input the slice graph structure into the multi-layer perceptron layer to obtain the output initial graph structure features;

[0160] Extracting features from the initial graph structure features by connecting at least one graph feature extraction layer in series to obtain at least one hierarchical graph structure feature as output;

[0161] Inputting the initial graph structure feature and at least one hierarchical graph structure feature into the splicing layer to obtain an output splicing graph structure feature;

[0162] Input the spliced ​​graph structure features into the graph structure encoding layer for feature encoding to obtain the output slice encoding features;

[0163] Among them, the input data of the first graph feature extraction layer is the initial graph structure feature, and the input data of the graph feature extraction layers other than the first graph feature extraction layer is the hierarchical graph structure feature output by the previous image feature extraction layer.

[0164] In an optional embodiment, the graph feature extraction layer includes a graph attention module, a graph convolution module and a fusion module;

[0165] Among them, the graph attention module is used to perform graph attention processing on the input data to obtain the output graph attention features, the graph convolution module is used to perform graph convolution processing on the input data to obtain the output graph convolution features, and the fusion module is used to perform feature fusion on the graph attention features and the graph convolution features to obtain the output hierarchical graph structure features.

[0166] In an optional embodiment, the graph convolution module includes at least two graph convolution units with residual connections, wherein the input data of the first graph convolution unit is the input data of the graph convolution module, the input data of the graph convolution units other than the first graph convolution unit are the input data and output results of the previous graph convolution unit, and the output result of the last graph convolution unit is the graph convolution feature.

[0167] In an optional embodiment, the image feature data includes scan image features and / or radiomics features corresponding to a tomographic image of the target object, and the apparatus further includes:

[0168] A scanning image feature determination module, used to perform image segmentation training on the encoder-decoder model based on the tomographic image;

[0169] The tomographic image is input into the encoder in the trained encoder-decoder model to obtain the scanned image features corresponding to the tomographic image output by the encoder.

[0170] The medical data classification device provided in the embodiment of the present invention can execute the medical data classification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0171] Figure 8 A schematic diagram of the structure of an electronic device provided by one embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0172] like Figure 8 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor 11, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0173] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information or data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0174] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the classification method of medical data provided in the above embodiments.

[0175] In some embodiments, the classification method of medical data provided in the above embodiments may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps in the classification method of medical data described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the classification method of medical data in any other appropriate manner (e.g., by means of firmware).

[0176] Various embodiments of the systems and techniques described above herein may be implemented in the following systems or combinations thereof: digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0177] The computer program for implementing the classification method of medical data of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0178] In the context of the present application, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable storage medium. Examples of machine-readable storage media may include electrical connections based on at least one line, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM), flash memories, optical fibers, portable compact disc read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0179] To provide interaction with a user, the systems and techniques described herein may be implemented on a terminal device having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the terminal device. Other types of devices may also provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0180] The systems and techniques described herein may be implemented in a computing system that includes a backend component (e.g., as a data server), or a computing system that includes a middleware component (e.g., an application server), or a computing system that includes a frontend component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), a blockchain network, and the Internet.

[0181] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0182] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0183] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for classifying medical data, characterized in that: include: Inputting the multimodal medical data of the target object into a trained medical classification model; wherein the multimodal medical data includes at least two modal data of image feature data, full-slice feature data, indicator parameter data, and gene sequencing data, and the medical classification model includes a classification output network and a feature extraction network corresponding to the at least two modal data respectively; For each type of modal data, the modal data is input into a corresponding feature extraction network to obtain an output modal feature vector; At least two modality feature vectors are input into the classification output network to obtain classification labels corresponding to the output multimodal medical data.

2. The method according to claim 1, characterized in that The full-slice feature data includes a slice graph structure and / or slice omics features corresponding to the full-slice image of the target object. Accordingly, the method further includes: Segmenting the full slice image of the target object to obtain a slice image block set; The abnormal tissue recognition model is trained according to the slice image block set, and the image block feature set is determined according to the abnormal tissue recognition model obtained by training; wherein the image block feature set includes image block features corresponding to each slice image block in the slice image block set; Performing similarity screening on the image block feature set to obtain a screened image block feature set, and determining a slice graph structure based on the screened image block feature set; The nodes in the slice graph structure represent the image block features, and the edges between the nodes represent that the distance between two slice image blocks is less than a distance threshold.

3. The method according to claim 2, characterized in that When the full-slice feature data includes a slice graph structure and a slice omics feature, the feature extraction network corresponding to the full-slice feature data includes a graph feature encoding layer, a bilinear layer, and an attention layer; The step of inputting the modal data into a corresponding feature extraction network to obtain an output modal feature vector comprises: Inputting the slice graph structure into the graph feature coding layer for feature coding to obtain output slice coding features; Inputting the slice coding feature and the slice omics feature into the bilinear layer for linear fusion to obtain an output slice fusion feature; The slice fusion feature is input into the attention layer for attention processing to obtain the modal feature vector corresponding to the output full slice feature data.

4. The method according to claim 3, characterized in that The graph feature encoding layer includes a multi-layer perceptron, at least one graph feature extraction layer, a splicing layer and a graph structure encoding layer; The step of inputting the slice graph structure into the graph feature coding layer for feature coding to obtain output slice coding features includes: Inputting the slice graph structure into the multi-layer perceptron layer to obtain output initial graph structure features; By connecting at least one graph feature extraction layer in series, extracting the initial graph structure feature to obtain at least one hierarchical graph structure feature as output; Inputting the initial graph structure feature and at least one hierarchical graph structure feature into the splicing layer to obtain an output splicing graph structure feature; Inputting the spliced ​​graph structure features into the graph structure encoding layer for feature encoding to obtain output slice encoding features; The input data of the first graph feature extraction layer is the initial graph structure feature, and the input data of the graph feature extraction layers other than the first graph feature extraction layer is the hierarchical graph structure feature output by the previous image feature extraction layer.

5. The method according to claim 4, characterized in that The graph feature extraction layer includes a graph attention module, a graph convolution module and a fusion module; Among them, the graph attention module is used to perform graph attention processing on the input data to obtain output graph attention features, the graph convolution module is used to perform graph convolution processing on the input data to obtain output graph convolution features, and the fusion module is used to perform feature fusion on the graph attention features and the graph convolution features to obtain output hierarchical graph structure features.

6. The method according to claim 5, characterized in that The graph convolution module includes at least two graph convolution units with residual connections, wherein the input data of the first graph convolution unit is the input data of the graph convolution module, the input data of the graph convolution units other than the first graph convolution unit are the input data and output results of the previous graph convolution unit, and the output result of the last graph convolution unit is the graph convolution feature.

7. The method according to claim 1, characterized in that The image feature data includes scan image features and / or radiomics features corresponding to the tomographic image of the target object. Accordingly, the method further includes: Performing image segmentation training on an encoder-decoder model based on the tomographic image; The tomographic image is input into the encoder in the trained encoder-decoder model to obtain the scanned image features corresponding to the tomographic image output by the encoder.

8. A medical data classification device, characterized in that: include: A multimodal medical data input module, used to input the multimodal medical data of the target object into a trained medical classification model; wherein the multimodal medical data includes at least two modal data of image feature data, full-slice feature data, indicator parameter data and gene sequencing data, and the medical classification model includes a classification output network and a feature extraction network corresponding to the at least two modal data respectively; A modal feature vector output module, for each type of modal data, inputs the modal data into a corresponding feature extraction network to obtain an output modal feature vector; The classification label output module is used to input at least two modality feature vectors into the classification output network to obtain the classification labels corresponding to the output multimodal medical data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the medical data classification method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the medical data classification method according to any one of claims 1 to 7 when executed.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the medical data classification method according to any one of claims 1 to 7.

Images