A multimodal fusion recognition system and device for pulmonary granulomatous mycobacterium infection

Abstract

The present invention provides a multimodal fusion recognition system and device for pulmonary granulomatous mycobacterium infection, which relates to the technical field of multimodal fusion recognition. The system mainly comprises: a pathological image feature extraction module, a CT image feature extraction module, a clinical information feature extraction module and a feature fusion classification module; the pathological image feature extraction module comprises a preprocessing unit, a segmentation unit and a pathological feature unit; the feature fusion classification module comprises a self-attention mechanism module, which is used to input the pathological features, the CT image features and the clinical information features into the self-attention mechanism module, and output the recognition result of pulmonary granulomatous mycobacterium infection; this scheme comprehensively integrates pulmonary granulomatous disease information of multiple modalities such as pathology, imaging and clinical, and can accurately, simply and economically identify pulmonary granulomatous mycobacterium infection.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal fusion recognition, and in particular to a multimodal fusion recognition system and device for pulmonary granulomatous mycobacterium infection. Background Art

[0002] Pulmonary granulomatous disease refers to a general term for lung diseases with granulomatous lesions or granulomatous lesions as the main pathological morphology caused by various reasons. It is a highly heterogeneous disease. According to its different causes, it can be divided into two categories: infectious and non-infectious. Among them, infectious granulomas mainly include non-mycobacterial infections and mycobacterial infections. Mycobacterial infections mainly include non-tuberculous mycobacterial infections and Mycobacterium tuberculosis infections. Among them, the pathological changes of Mycobacterium tuberculosis are very complex due to its virulence, the amount of infected bacteria, and the body's own immunity. Nontuberculous mycobacteria (NTM) are not only numerous in variety (more than 190 species have been discovered), but also overlap and cross with Mycobacterium tuberculosis, which makes the characteristics of mycobacteria in pathology or imaging more complex.

[0003] Although pulmonary granulomatous diseases of different etiologies have some characteristics in pathology and imaging, they are mostly non-specific manifestations. Therefore, only a single type of data is not enough to confirm whether it is a mycobacterial infection. At the same time, the clinical and imaging manifestations of pulmonary granulomas of different etiologies are slightly different, and it is difficult to accurately assess the cause by relying solely on microscopic morphological observations. The cultivation and species identification of mycobacteria usually takes a long time, generally about 2-6 weeks; a series of complex biochemical tests and molecular biological tests are also required to determine its species: among them, biochemical tests are not only complex to operate, the results are unstable and time-consuming; molecular biology tests are not only costly, technically difficult, and have a long experimental cycle, but also require high sample quality. Summary of the invention

[0004] The object of the present invention is to provide a multimodal fusion identification system and device for pulmonary granulomatous mycobacterium infection to solve at least one of the above-mentioned technical problems existing in the prior art.

[0005] In the first aspect, in order to solve the above technical problems, the present invention provides a multimodal fusion recognition system for pulmonary granulomatous mycobacterium infection, which mainly includes a pathological image feature extraction module, a CT image feature extraction module, a clinical information feature extraction module and a feature fusion classification module;

[0006] The pathological image feature extraction module includes a preprocessing unit, a segmentation unit and a pathological feature unit;

[0007] The preprocessing unit is used to receive the pathological image of the pulmonary granuloma case and perform preprocessing;

[0008] The segmentation unit segments the tissue area of ​​the preprocessed pathological image based on the segmentation threshold; divides the tissue area into patches (i.e., image blocks) of a preset size to obtain slices;

[0009] The pathological feature unit inputs the patch block into a self-supervised model (such as a DINO model), extracts patch features, and then forms a patch package with all the patch blocks in a slice, which is input into a first attention module (such as a GAMIL module, a gated attention multi-instance model), assigns a first weight value according to the importance of each patch block (the greater the contribution of the patch block to the classification, the higher the assigned first weight value), multiplies the patch feature with the corresponding first weight value, and outputs a WSI global feature vector; at the same time, the patch blocks are respectively input into the supervision models corresponding to different tissue regions, and the WSI feature vectors of the corresponding tissue regions are output; the WSI global feature vector and the WSI feature vector of the corresponding tissue region are stacked into a feature package, which is input into a second attention module (such as a GAMIL module) to obtain pathological features; in this way, the full slice information and the key focus area information can be fully integrated, and the relationship between different features and pulmonary granulomatous mycobacterium infection can be automatically learned through the GAMIL module;

[0010] The CT image feature extraction module is used to receive CT image data of pulmonary granuloma cases, and extract CT image features through a Transformer-based 3D neural network model to obtain specific distribution characteristics of granulomas, including: distribution along lymphatic channels, distribution along airways, and random distribution, so as to help distinguish the types of pulmonary granuloma diseases in the future; the 3D neural network model includes an encoder, a decoder, and a classifier; in this way, the multiple self-attention mechanism of the Transformer encoder can effectively capture long-range dependencies, thereby better capturing CT image features;

[0011] The clinical information feature extraction module is used to receive clinical information of pulmonary granuloma cases, obtain numerical features of the clinical information through One-hot encoding, amplify the numerical features through linear feature mapping, and then learn the amplified numerical features through a shallow attention mechanism encoder to obtain clinical information features;

[0012] The feature fusion classification module includes a self-attention mechanism module, which is used to input the pathological features, the CT image features and the clinical information features into the self-attention mechanism module, and output the recognition results of pulmonary granulomatous mycobacterium infection; the self-attention mechanism module is divided into three branches from top to bottom: query q, key value k and value v; compared with the traditional attention mechanism, this scheme can calculate the degree of association between each feature element and all other elements, thereby establishing a global dependency relationship, expanding the receptive field, and more effectively integrating image, pathological and clinical features.

[0013] Through the above system, this scheme can automatically and effectively fuse multimodal data features, and identify pulmonary granulomatous mycobacterium infection simply, economically and reliably, thereby avoiding the problems of complex biochemical test operations, unstable results and long time consumption. It also avoids the problems of high cost, high technical difficulty, long experimental cycle and high sample quality requirements of molecular biology testing.

[0014] In a feasible implementation, the preprocessing in the preprocessing unit specifically includes: in the RGB color space, the pathological image is segmented by the OTSU threshold method to obtain a first mask image mask_1; in the S channel of the HSV color space, the pathological image is segmented by the OTSU threshold method to obtain a second mask image mask_2; mask_1 and mask_2 are bitwise ANDed to obtain a third mask image mask; the bitwise AND operation means that when the corresponding bits in the two images are both true, the value is true.

[0015] In a feasible implementation manner, the size of the patch block is 512*512.

[0016] In a feasible implementation manner, the size of the patch package is N*M, where N represents the number of blocks in the patch, and M represents the feature dimension of the patch.

[0017] In a feasible implementation, the calculation method of the second attention module includes: receiving a feature vector F (size is N*M) of a slice, first upsampling the feature vector F through linear mapping to obtain an upsampled feature vector F1 of size N*M1 (M1>M); then passing through the first activation function layer; then downsampling the feature vector F through linear mapping to obtain a downsampled feature vector F2 of size N*M2 (M2<M1); then passing through the second activation function layer; then through linear mapping to obtain a second weight vector W of size N*1; then multiplying the feature vector F by the second weight vector W to obtain the pathological feature.

[0018] In a feasible implementation, the supervision model includes a granuloma recognition model, a necrosis recognition model, a lymphocyte cluster recognition model, a multinucleated giant cell recognition model, etc.;

[0019] The granuloma recognition model is used to obtain the tissue features of the granuloma from the patch block; then input it into the third attention module (such as the GAMIL module) to output the granuloma WSI (Whole Slide Image, i.e., pathological section) feature vector;

[0020] The necrosis recognition model is used to obtain necrotic tissue features from the patch block; then input it into the fourth attention module (such as the GAMIL module) to output the necrotic WSI feature vector;

[0021] The lymphocyte cluster recognition model is used to obtain the tissue characteristics of the lymphocyte cluster from the patch block; then input it into the fifth attention module (such as the GAMIL module) to output the lymphocyte cluster WSI feature vector;

[0022] The multinuclear giant cell recognition model is used to obtain the tissue features of the multinuclear giant cells from the patch block; then input it into the sixth attention module (such as the GAMIL module) to output the multinuclear giant cell WSI feature vector;

[0023] In this way, the tissue features of granuloma areas, necrosis areas, lymphocyte cluster areas and multinuclear giant cell areas can be effectively extracted, thereby taking into account all important tissue areas in the pathological image, and thus ensuring the reliability of subsequent feature fusion classification.

[0024] In a feasible implementation, the encoding method of the encoder in the 3D neural network model includes: receiving a CT sequence (size is H*W*D*4, D represents the sequence length, W represents the width of the CT image, and H represents the height of the CT image), and inputting it into a first-level encoder layer; dividing the CT sequence into non-overlapping T (T=[H / m]×[W / m]×[D / p]) 3D sequence blocks through 3D convolution in the first-level encoder layer (the convolution kernel size is m*m*p, p represents the thickness of the convolution kernel, and m represents the length and width of the convolution kernel); mapping each 3D sequence block (dimension is m*m*p) to a C-dimensional vector through linear embedding of the first-level downsampling layer; and then mapping the T C The dimensional vector is input into the next encoder layer, and 3D convolution and linear embedding operations are performed cyclically until the fourth encoder layer; the encoder layer includes two sub-layers, namely a multi-head self-attention mechanism (MSA) and a fully connected feedforward network (MLP), and these sub-layers are interconnected through residual connections and LayerNorm (LN) normalization, respectively; in this way, the features of the CT sequence can be gradually extracted through the encoder layer, and the relationship between each position in the CT sequence can be captured through the multi-head self-attention mechanism. Compared with the simple use of convolutional neural networks, which only extracts image features layer by layer through convolution operations and pooling operations, this scheme can obtain a larger receptive field in the shallow layer, thereby improving learning efficiency and effect.

[0025] In a feasible implementation, the decoding method of the decoder in the 3D neural network model includes: each level of the decoder layer simultaneously receives the output features of the same level encoder layer and the previous level decoder layer; after stacking the output features of these two parts, they are input into the 3D network layer for upsampling operation.

[0026] In a feasible implementation, the classifier in the 3D neural network model includes a 3D convolutional layer for mapping the C-dimensional vector to K segmentation classes, and calculating the loss function according to the segmentation results, thereby optimizing the 3D neural network model. The segmentation classes include lesion classes related to pulmonary granuloma, such as boundaries, calcifications, and ground-glass opacities.

[0027] In a feasible implementation, the loss function includes soft Dice loss and / or cross entropy loss;

[0028] The specific formula of the soft Dice loss function is:

[0029] ;

[0030] The specific formula of the cross entropy loss function is:

[0031] ;

[0032] Among them, p represents the predicted probability of the 3D neural network model, and t represents the true label.

[0033] In a feasible implementation, the 3D neural network model further includes combining output features of decoder layers at each level as CT image features.

[0034] In a feasible implementation, the clinical information includes fever, symptoms of tuberculosis poisoning (fatigue, night sweats, weight loss, etc.), cough, hemoptysis, history of tuberculosis contact, and scoliosis, superficial lymphadenopathy, subcutaneous nodules, tattoos, arrhythmia, hepatosplenomegaly, and edema of both lower limbs;

[0035] For example, if the case has fever symptoms, the corresponding information is set to 1; if there are no fever symptoms, the corresponding information is set to 0.

[0036] In a feasible implementation, the amplification also includes introducing noise and / or data transformation; the introduction of noise specifically refers to randomly inserting a number of noise points into the data; the data transformation refers to amplifying the data through linear function mapping; this can enhance the robustness and stability of the clinical information feature extraction module, and better adapt to the situation where the amount of effective clinical information data is relatively small and the amount of abnormal data is relatively large.

[0037] In a feasible implementation, the specific calculation method of the self-attention mechanism module includes:

[0038] Step a1: After obtaining the query q, similarity calculation is performed with each key value k to obtain the third weight. The specific formula can be:

[0039] ;

[0040] in, Indicates The query and The third weight of the similarity between key values; Represents the vector dimension of key value k; Indicates The specific formula is:

[0041] ;

[0042] in, A trainable parameter matrix representing the query q; Indicates Input matrix;

[0043] Indicates key value, the specific formula is:

[0044] ;

[0045] in, A trainable parameter matrix representing the key value k; Indicates Input matrix;

[0046] Step a2: normalize the third weight by using the softmax function. The specific formula can be:

[0047] ;

[0048] in, express The normalized third weight of ;

[0049] Step a3: perform weighted summation of the normalized third weight and the value v corresponding to the key value k to obtain the final feature, input it into the linear classifier, and output the recognition result of pulmonary granulomatous mycobacterium infection; the recognition result is: yes or no; the specific calculation formula is:

[0050] ;

[0051] in, Indicates The final feature; Indicates The specific formula can be:

[0052] ;

[0053] in, A trainable parameter matrix representing the value v;

[0054] It should be noted that the input matrix is ​​respectively , and Multiplying the parameters of these three matrices is equivalent to performing a linear transformation, and we get the query, key value, and value.

[0055] In the second aspect, based on the same inventive concept, the present application also provides a multimodal fusion identification device for pulmonary granulomatous Mycobacterium infection, including a processor, a memory and a bus, wherein the memory stores instructions and data readable by the processor, and the processor is used to call the instructions and data in the memory to implement the multimodal fusion identification system for pulmonary granulomatous Mycobacterium infection as described above, and the bus connects the functional components for transmitting information.

[0056] By adopting the above technical solution, the present invention has the following beneficial effects:

[0057] The multimodal fusion identification system and device for pulmonary granulomatous mycobacterium infection provided by the present invention fully integrates the information of pulmonary granulomatous diseases in multiple modalities such as pathology, imaging and clinical, and can identify pulmonary granulomatous mycobacterium infection more accurately, simply and economically. Moreover, the network structure designed in this scheme can make full use of global features, avoiding the problem of extracting image features layer by layer through convolution and pooling operations when using a convolutional neural network alone, resulting in a small feature receptive field and lack of globality when the shallow layer is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A layout diagram of a multimodal fusion recognition system for pulmonary granulomatous mycobacterium infection provided by an embodiment of the present invention;

[0060] Figure 2 A diagram illustrating the working principle of a multimodal fusion recognition system for pulmonary granulomatous mycobacterium infection provided by an embodiment of the present invention;

[0061] Figure 3 A diagram illustrating the working principle of a pathological image feature extraction module provided in an embodiment of the present invention;

[0062] Figure 4 A diagram illustrating the working principle of a CT image feature extraction module provided in an embodiment of the present invention;

[0063] Figure 5 A flowchart of a specific calculation method for the self-attention mechanism module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0066] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0067] The present invention is further explained below in conjunction with specific implementation modes.

[0068] It should also be noted that the following specific embodiments or specific implementations are a series of optimized settings listed in the present invention to further explain the specific content of the invention, and these settings can be used in combination or in association with each other.

[0069] Embodiment 1:

[0070] like Figure 1-2 As shown, the multimodal fusion recognition system for pulmonary granulomatous mycobacterium infection provided in this embodiment mainly includes a pathological image feature extraction module, a CT image feature extraction module, a clinical information feature extraction module and a feature fusion classification module;

[0071] The pathological image feature extraction module includes a preprocessing unit, a segmentation unit and a pathological feature unit;

[0072] The preprocessing unit is used to receive the pathological image of the pulmonary granuloma case and perform preprocessing;

[0073] The segmentation unit segments the tissue area of ​​the preprocessed pathological image based on the segmentation threshold; divides the tissue area into patches of a preset size to obtain slices;

[0074] The pathological feature unit inputs the patch block into a self-supervised model (such as a DINO model), extracts patch features, and then forms a patch package with all the patch blocks in a slice, which is input into a first attention module (such as a GAMIL module), and after assigning different first weight values ​​according to the importance of each patch block (the greater the contribution of the patch block to the classification, the higher the assigned first weight value), multiplies the patch features with the corresponding first weight values, and outputs a WSI global feature vector; at the same time, the patch blocks are respectively input into the supervision models corresponding to different tissue regions, and the WSI feature vectors of the corresponding tissue regions are output; the WSI global feature vector and the WSI feature vector of the corresponding tissue region are stacked into a feature package, which is input into a second attention module (such as a GAMIL module) to obtain pathological features; in this way, the full slice information and the key focus area information can be fully integrated, and the relationship between different features and pulmonary granulomatous mycobacterium infection can be automatically learned through the GAMIL module;

[0075] The CT image feature extraction module is used to receive CT image data of pulmonary granuloma cases, and extract CT image features through a Transformer-based 3D neural network model to obtain specific distribution characteristics of granulomas, including: distribution along lymphatic channels, distribution along airways, and random distribution, so as to help distinguish the types of pulmonary granuloma diseases in the future; the 3D neural network model includes an encoder, a decoder, and a classifier; in this way, the multiple self-attention mechanism of the Transformer encoder can effectively capture long-range dependencies, thereby better capturing CT image features;

[0076] The clinical information feature extraction module is used to receive clinical information of pulmonary granuloma cases, obtain numerical features of the clinical information through One-hot encoding, amplify the numerical features through linear feature mapping, and then learn the amplified numerical features through a shallow attention mechanism encoder to obtain clinical information features;

[0077] The feature fusion classification module includes a self-attention mechanism module, which is used to input the pathological features, the CT image features and the clinical information features into the self-attention mechanism module, and output the recognition results of pulmonary granulomatous mycobacterium infection; the self-attention mechanism module is divided into three branches from top to bottom: query q, key value k and value v; compared with the traditional attention mechanism, this scheme can calculate the degree of association between each feature element and all other elements, thereby establishing a global dependency relationship, expanding the receptive field, and more effectively integrating image, pathological and clinical features.

[0078] Through the above system, multimodal data features can be automatically and effectively integrated to identify pulmonary granulomatous mycobacterium infection simply, economically and reliably, thus avoiding the problems of complex biochemical test operations, unstable results and long time consumption. It also avoids the problems of high cost, high technical difficulty, long experimental cycle and high sample quality requirements of molecular biology testing.

[0079] Furthermore, the preprocessing in the preprocessing unit specifically includes: in the RGB color space, the pathological image is segmented by the OTSU threshold method to obtain a first mask image mask_1; in the S channel of the HSV color space, the pathological image is segmented by the OTSU threshold method to obtain a second mask image mask_2; mask_1 and mask_2 are bitwise ANDed to obtain a third mask image mask; the bitwise AND operation means that when the corresponding bits in the two images are both true, the value is true.

[0080] Furthermore, the size of the patch block is 512*512.

[0081] Furthermore, the size of the patch package is N*M; N represents the number of blocks in the patch, and M represents the feature dimension of the patch.

[0082] Furthermore, the calculation method of the second attention module includes: receiving a feature vector F (size is N*M) of a slice, first upsampling the feature vector F through linear mapping to obtain an upsampled feature vector F1 of size N*M1 (M1>M); then passing through the first activation function layer; then downsampling the feature vector F through linear mapping to obtain a downsampled feature vector F2 of size N*M2 (M2<M1); then passing through the second activation function layer; then through linear mapping to obtain a second weight vector W of size N*1; then multiplying the feature vector F by the second weight vector W to obtain the pathological feature.

[0083] Furthermore, if Figure 3 As shown, the supervised model includes a granuloma recognition model, a necrosis recognition model, a lymphocyte cluster recognition model and a multinuclear giant cell recognition model;

[0084] The granuloma recognition model is used to obtain the tissue features of the granuloma from the patch block; then input it into the third attention module (such as the GAMIL module) to output the granuloma WSI feature vector;

[0085] The necrosis recognition model is used to obtain necrotic tissue features from the patch block; then input it into the fourth attention module (such as the GAMIL module) to output the necrotic WSI feature vector;

[0086] The lymphocyte cluster recognition model is used to obtain the tissue characteristics of the lymphocyte cluster from the patch block; then input it into the fifth attention module (such as the GAMIL module) to output the lymphocyte cluster WSI feature vector;

[0087] The multinuclear giant cell recognition model is used to obtain the tissue features of the multinuclear giant cells from the patch block; then input it into the sixth attention module (such as the GAMIL module) to output the multinuclear giant cell WSI feature vector;

[0088] In this way, the tissue features of granuloma areas, necrosis areas, lymphocyte cluster areas and multinuclear giant cell areas can be effectively extracted, thereby taking into account all important tissue areas in the pathological image, and thus ensuring the reliability of subsequent feature fusion classification.

[0089] Furthermore, if Figure 4 As shown, the encoding method of the encoder in the 3D neural network model includes: receiving a CT sequence (size is H*W*D*4, D represents the sequence length, W represents the width of the CT image, and H represents the height of the CT image), and inputting it into the first-level encoder layer; dividing the CT sequence into non-overlapping T (T=[H / m]×[W / m]×[D / p]) 3D sequence blocks through 3D convolution in the first-level encoder layer (the convolution kernel size is m*m*p, p represents the thickness of the convolution kernel, and m represents the length and width of the convolution kernel); mapping each 3D sequence block (dimension is m*m*p) to a C-dimensional vector through linear embedding of the first-level downsampling layer; and then mapping T The C-dimensional vectors are input into the next encoder layer, and 3D convolution and linear embedding operations are performed cyclically until the fourth encoder layer; the encoder layer includes two sublayers, namely a multi-head self-attention mechanism and a fully connected feedforward network, and these sublayers are interconnected through residual connections and LayerNorm normalization, respectively; in this way, the features of the CT sequence can be gradually extracted through the encoder layer, and the relationship between each position in the CT sequence can be captured through the multi-head self-attention mechanism. Compared with the simple use of convolutional neural networks, which only extracts image features layer by layer through convolution operations and pooling operations, this scheme can obtain a larger receptive field in a shallow layer, thereby improving learning efficiency and effect.

[0090] Furthermore, the decoding method of the decoder in the 3D neural network model includes: each level of the decoder layer simultaneously receives the output features of the same level encoder layer and the previous level decoder layer; after stacking the output features of these two parts, they are input into the 3D network layer for upsampling operation.

[0091] Furthermore, the classifier in the 3D neural network model includes a 3D convolutional layer for mapping the C-dimensional vector to K segmentation classes, and calculating the loss function according to the segmentation results, thereby optimizing the 3D neural network model. The segmentation classes include lesion classes related to pulmonary granuloma, such as enchantments, calcifications, and ground-glass opacities.

[0092] Further, the loss function includes soft Dice loss and cross entropy loss;

[0093] The specific formula of the soft Dice loss function is:

[0094] ;

[0095] The specific formula of the cross entropy loss function is:

[0096] ;

[0097] Among them, p represents the predicted probability of the 3D neural network model, and t represents the true label.

[0098] Furthermore, the 3D neural network model also includes combining output features of decoder layers at each level as CT image features.

[0099] Furthermore, the clinical information includes fever, symptoms of tuberculosis poisoning (fatigue, night sweats and weight loss, etc.), cough, hemoptysis, history of tuberculosis contact and scoliosis, superficial lymphadenopathy, subcutaneous nodules, tattoos, arrhythmia, hepatosplenomegaly and edema of both lower limbs;

[0100] For example, if the case has fever symptoms, the corresponding information is set to 1; if there are no fever symptoms, the corresponding information is set to 0.

[0101] Furthermore, the amplification includes introducing noise or data transformation; the introduction of noise specifically refers to randomly inserting a number of noise points into the data; the data transformation refers to amplifying the data through linear function mapping; this can enhance the robustness and stability of the clinical information feature extraction module, and better adapt to the situation where the amount of effective clinical information data is relatively small and the amount of abnormal data is relatively large.

[0102] Furthermore, if Figure 5 As shown, the specific calculation method of the self-attention mechanism module includes:

[0103] Step a1: After obtaining the query q, similarity calculation is performed with each key value k to obtain the third weight. The specific formula can be:

[0104] ;

[0105] in, Indicates The query and The third weight of the similarity between key values; Represents the vector dimension of key value k; Indicates The specific formula is:

[0106] ;

[0107] in, A trainable parameter matrix representing the query q; Indicates Input matrix;

[0108] Indicates key value, the specific formula is:

[0109] ;

[0110] in, A trainable parameter matrix representing the key value k; Indicates Input matrix;

[0111] Step a2: normalize the third weight by using the softmax function. The specific formula can be:

[0112] ;

[0113] in, express The normalized third weight of ;

[0114] Step a3: perform weighted summation of the normalized third weight and the value v corresponding to the key value k to obtain the final feature, input it into the linear classifier, and output the recognition result of pulmonary granulomatous mycobacterium infection; the recognition result is: yes or no; the specific calculation formula is:

[0115] ;

[0116] in, Indicates The final feature; Indicates The specific formula can be:

[0117] ;

[0118] in, A trainable parameter matrix representing the value v;

[0119] It should be noted that the input matrix is ​​respectively , and Multiplying the parameters of these three matrices is equivalent to performing a linear transformation, and we get the query, key value, and value.

[0120] Embodiment 2:

[0121] The present embodiment provides a multimodal fusion identification device for pulmonary granulomatous mycobacterium infection, including a processor, a memory and a bus. The memory stores instructions and data that can be read by the processor. The processor is used to call the instructions and data in the memory to implement the multimodal fusion identification system for pulmonary granulomatous mycobacterium infection as described above. The bus connects the functional components for transmitting information.

[0122] In another embodiment, the present solution can be implemented by an integrated device, which may include corresponding modules for performing each or several steps in the above-mentioned embodiments. The module may be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0123] The processor performs the various methods and processes described above. For example, the method implementation in the present solution can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be executed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods by any other appropriate means (for example, by means of firmware).

[0124] The device can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus connects various circuits including one or more processors, memories, and / or hardware modules together. The bus can also connect various other circuits such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0125] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal fusion recognition system for pulmonary granulomatous mycobacterium infection, characterized in that: It includes pathological image feature extraction module, CT image feature extraction module, clinical information feature extraction module and feature fusion classification module; The pathological image feature extraction module includes a preprocessing unit, a segmentation unit and a pathological feature unit; The preprocessing unit is used to receive the pathological image of the pulmonary granuloma case and perform preprocessing; The segmentation unit segments the tissue area of ​​the preprocessed pathological image based on the segmentation threshold; divides the tissue area into patches of a preset size to obtain slices; The pathological feature unit inputs the patch block into the self-supervised model, extracts patch features, and then assembles all patch blocks in a slice into a patch package, which is input into the first attention module. After assigning a first weight value according to the importance of each patch block, the patch feature is multiplied by the corresponding first weight value, and a WSI global feature vector is output; at the same time, the patch blocks are respectively input into the supervision models corresponding to different tissue regions, and the WSI feature vectors of the corresponding tissue regions are output; The WSI global feature vector and the WSI feature vector of the corresponding tissue area are stacked into a feature bag, which is input into the second attention module to obtain the pathological features; The CT image feature extraction module is used to receive CT image data of pulmonary granuloma cases and extract CT image features through a Transformer-based 3D neural network model; the 3D neural network model includes an encoder, a decoder and a classifier; The clinical information feature extraction module is used to receive clinical information of pulmonary granuloma cases and obtain numerical features of the clinical information through One-hot encoding; Then, the numerical features are amplified through linear feature mapping; and then, the amplified numerical features are learned through a shallow attention mechanism encoder to obtain clinical information features; The feature fusion classification module includes a self-attention mechanism module, which is used to input the pathological features, the CT image features and the clinical information features into the self-attention mechanism module and output the recognition result of pulmonary granulomatous mycobacterium infection.

2. The system according to claim 1, characterized in that The preprocessing in the preprocessing unit specifically includes: in the RGB color space, the pathological image is segmented by the OTSU threshold method to obtain a first mask image mask_1; in the S channel of the HSV color space, the pathological image is segmented by the OTSU threshold method to obtain a second mask image mask_2; and mask_1 and mask_2 are bitwise ANDed to obtain a third mask image mask.

3. The system according to claim 1, characterized in that The size of the patch package is N*M; N represents the number of blocks in the patch, and M represents the feature dimension of the patch.

4. The system according to claim 3, characterized in that The calculation method of the second attention module includes: receiving a feature vector F of a slice, first upsampling the feature vector F through linear mapping to obtain an upsampled feature vector F1 of size N*M1; then passing through the first activation function layer; then downsampling the feature vector F through linear mapping to obtain a downsampled feature vector F2 of size N*M2; then passing through the second activation function layer; then through linear mapping to obtain a second weight vector W of size N*1; then multiplying the feature vector F by the second weight vector W to obtain the pathological feature.

5. The system according to claim 1, characterized in that The supervision model includes a granuloma recognition model, a necrosis recognition model, a lymphocyte cluster recognition model and a multinuclear giant cell recognition model; The granuloma recognition model is used to obtain the tissue features of the granuloma from the patch block; then input it into the third attention module to output the granuloma WSI feature vector; the WSI represents the pathological section; The necrosis recognition model is used to obtain necrotic tissue features from the patch block; then input it into the fourth attention module to output the necrotic WSI feature vector; The lymphocyte cluster recognition model is used to obtain the tissue characteristics of the lymphocyte cluster from the patch block; then input it into the fifth attention module to output the lymphocyte cluster WSI feature vector; The multinuclear giant cell recognition model is used to obtain the tissue characteristics of multinuclear giant cells from the patch block; then it is input into the sixth attention module to output the multinuclear giant cell WSI feature vector.

6. The system according to claim 1, characterized in that The encoding method of the encoder in the 3D neural network model includes: receiving a CT sequence and inputting it into a first-level encoder layer; dividing the CT sequence into T non-overlapping 3D sequence blocks through 3D convolution in the first-level encoder layer; mapping each 3D sequence block to a C-dimensional vector through linear embedding in the first-level downsampling layer; then inputting the T C-dimensional vectors into the next-level encoder layer, and cyclically performing 3D convolution and linear embedding operations until the fourth-level encoder layer; the encoder layer includes two sublayers, namely a multi-head self-attention mechanism and a fully connected feedforward network, and these sublayers are interconnected through residual connections and LayerNorm normalization, respectively.

7. The system according to claim 6, characterized in that The decoding method of the decoder in the 3D neural network model includes: each level of the decoder layer simultaneously receives the output features of the same level encoder layer and the previous level decoder layer; after stacking the output features of these two parts, they are input into the 3D network layer for upsampling operation.

8. The system according to claim 7, characterized in that The classifier in the 3D neural network model includes a 3D convolution layer, which is used to map the C-dimensional vector to K segmentation classes and calculate the loss function according to the segmentation results.

9. The system according to claim 1, characterized in that The specific calculation method of the self-attention mechanism module includes: Step a1: After obtaining the query q, perform similarity calculation with each key value k to obtain the third weight. The specific formula is: ； in, Indicates The query and The third weight of the similarity between key values; Represents the vector dimension of key value k; Indicates The specific formula is: ； in, represents the parameter matrix of query q; Indicates Input matrix; Indicates key value, the specific formula is: ； in, The parameter matrix representing the key value k; Indicates Input matrix; Step a2: Normalize the third weight through the softmax function. The specific formula is: ； in, express The normalized third weight of ; Step a3: perform weighted summation of the normalized third weight and the value v corresponding to the key value k to obtain the final feature, input it into the linear classifier, and output the recognition result of pulmonary granulomatous mycobacterium infection; the recognition result is: yes or no; the specific calculation formula is: ； in, Indicates The final feature; Indicates The specific formula is: ； in, A parameter matrix representing the values ​​v.

10. A multimodal fusion identification device for pulmonary granulomatous mycobacterium infection, characterized in that: It includes a processor, a memory and a bus, wherein the memory stores instructions and data read by the processor, the processor is used to call the instructions and data in the memory to implement the system as described in any one of claims 1-9, and the bus connects the functional components for transmitting information.

Images