Lung image segmentation method and device and storage medium

Abstract

The invention discloses a lung image segmentation method and apparatus, and a storage medium. The method comprises the steps of obtaining a lung image and radiology report information; preprocessing the lung image to obtain an initial image spatial feature and an initial image channel feature; preprocessing the radiology report information to obtain an initial space level text feature and an initial channel level text feature; performing multi-modal feature alignment processing by using the target cosine loss function to obtain a target alignment feature; constructing a space level multi-modal feature; multi-modal features of the channel level are constructed; according to a dual cross-modal self-attention mechanism, constructing a target multi-modal feature; and according to a preset semantic level number, performing adaptive gating fusion processing on the target multi-modal features on different semantic levels to obtain a lung image segmentation result. According to the method, lung image segmentation is realized, the accuracy and applicability are improved, and the calculation complexity is reduced. The method can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a lung image segmentation method, device and storage medium. Background Art

[0002] Text-assisted lung image segmentation is a technology that combines natural language processing (NLP) and medical image processing to identify lung images. Lung images themselves provide rich information about lung structure, but often lack intuitive semantic labels. Clinical medical reports usually contain detailed descriptions of lung images, which can provide support for image segmentation. Traditional lung image segmentation methods use a joint modeling approach to process image and text information at the same time, resulting in inconsistent feature spaces of different modal features, affecting the segmentation effect of cross-modal feature fusion, and low segmentation accuracy. At the same time, the text features of each word are repeatedly connected after the image features of each image pixel, resulting in high computational complexity. In addition, the model limits the length of text input, making it difficult to apply to a variety of application scenarios and having low applicability.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The embodiments of the present invention provide a lung image segmentation method, device and storage medium, which effectively improve the accuracy and applicability and reduce the computational complexity.

[0005] On the one hand, an embodiment of the present invention provides a lung image segmentation method, comprising the following steps:

[0006] obtain lung images and radiology report information;

[0007] Preprocessing the lung image to obtain initial image spatial features and initial image channel features;

[0008] Preprocessing the radiology report information to obtain initial spatial level text features and initial channel level text features;

[0009] According to the initial image spatial features, the initial image channel features, the initial spatial level text features and the initial channel level text features, a target cosine loss function is used to perform multimodal feature alignment processing to obtain a target alignment feature, wherein the target alignment feature includes a target image spatial feature, a target image channel feature, a target spatial level text feature and a target channel level text feature;

[0010] Constructing a spatial-level multimodal feature according to the target image spatial feature and the target spatial-level text feature;

[0011] Constructing a channel-level multimodal feature according to the target image channel feature and the target channel-level text feature;

[0012] Constructing a target multimodal feature according to the dual cross-modal self-attention mechanism, the spatial-level multimodal feature, and the channel-level multimodal feature;

[0013] According to the preset semantic level number, the target multimodal features at different semantic levels are adaptively gated and fused to obtain a lung image segmentation result.

[0014] In some embodiments, the preprocessing of the lung image to obtain initial image spatial features and initial image channel features includes:

[0015] extracting an initial feature map from the lung image;

[0016] extracting visual features from the initial feature map;

[0017] Performing L2 normalization on the visual features;

[0018] According to the feature map size, pixel coordinates and preset spatial coordinate feature dimensions, construct a spatial coordinate feature corresponding to each pixel in the initial feature map, wherein the spatial coordinate feature includes horizontal position information of a first vector dimension, vertical position information of a second vector dimension, image width information of a third vector dimension and image height information of a fourth vector dimension;

[0019] Performing L2 normalization on the spatial coordinate features;

[0020] Constructing the initial image spatial features according to the spatial coordinate features normalized by the L2 paradigm and the visual features normalized by the L2 paradigm;

[0021] The visual features normalized by the L2 paradigm are subjected to representation transformation processing to obtain the initial image channel features.

[0022] In some embodiments, the preprocessing of the radiology report information to obtain initial spatial level text features and initial channel level text features includes:

[0023] Performing word division processing on the radiology report information to obtain a word set;

[0024] One-hot encoding is performed on the word set to obtain multiple one-hot vectors;

[0025] Perform word embedding processing on the one-hot vector to obtain a word vector;

[0026] According to the image space feature dimension of the initial image space feature, performing a first linear transformation on the word vector so that the feature dimension of the word vector after the first linear transformation is consistent with the image space feature dimension;

[0027] Performing the L2 normalization on the word vector after the first linear transformation to obtain the initial spatial level text feature;

[0028] According to the image channel feature dimension of the initial image channel feature, performing a second linear transformation on the word vector, so that the feature dimension of the word vector after the second linear transformation is consistent with the image channel feature dimension;

[0029] The word vector after the second linear transformation is normalized using the L2 normal form to obtain the initial channel-level text features.

[0030] In some embodiments, the multimodal feature alignment process is performed using a target cosine loss function according to the initial image spatial features, the initial image channel features, the initial spatial level text features, and the initial channel level text features to obtain a target alignment feature, including:

[0031] Constructing an image spatial average feature according to the initial image spatial feature and the number of pixels in the feature map;

[0032] Constructing an image channel average feature according to the initial image channel feature and the number of feature map channels;

[0033] Constructing an average text feature at the spatial level according to the initial spatial level text feature and the number of text words;

[0034] Constructing a channel-level text average feature according to the initial channel-level text feature and the number of text words;

[0035] Constructing a spatial-level cosine loss function according to the spatial average feature of the image and the spatial-level text average feature;

[0036] Constructing a channel-level cosine loss function according to the image channel average feature and the channel-level text average feature;

[0037] Constructing the target cosine loss function according to the spatial level cosine loss function and the channel level cosine loss function;

[0038] The target cosine loss function is used to update the initial image space features, the initial image channel features, the initial space-level text features and the initial channel-level text features to obtain the target alignment features.

[0039] In some embodiments, constructing a spatial-level multimodal feature according to the target image spatial feature and the target spatial-level text feature includes:

[0040] Combining the target image spatial feature and the target spatial level text feature to obtain a spatial level cross-modal joint feature;

[0041] According to the spatial-level cross-modal joint features, a query vector, a key vector, and a value vector are constructed respectively;

[0042] constructing an attention score based on the query vector and the key vector;

[0043] Constructing a value weighted vector according to the attention score and the value vector;

[0044] According to the linear transformation weight, performing a third linear transformation on the value weighted vector so that the feature dimension of the value weighted vector is consistent with the feature dimension of the cross-modal joint feature at the spatial level;

[0045] Constructing a spatial-level cross-modal self-attention feature according to the value weighted vector after the third linear transformation and the spatial-level cross-modal joint feature;

[0046] The spatial-level multimodal features are constructed based on the spatial-level cross-modal self-attention features, and the channel dimension of the spatial-level multimodal features after the fourth linear transformation is 500 dimensions.

[0047] In some embodiments, constructing a channel-level multimodal feature according to the target image channel feature and the target channel-level text feature includes:

[0048] Combining the target image channel feature and the target channel-level text feature to obtain a channel-level cross-modal joint feature;

[0049] Constructing an attention map according to the channel-level cross-modal joint features;

[0050] Constructing a channel-level cross-modal self-attention feature according to the attention map and the channel-level cross-modal joint feature;

[0051] The channel-level multimodal features are constructed based on the channel-level cross-modal self-attention features, and the channel dimension of the channel-level multimodal features after the fifth linear transformation is 500 dimensions.

[0052] In some embodiments, the adaptive gating fusion processing is performed on the target multimodal features at different semantic levels according to the preset semantic level number to obtain the lung image segmentation result, including:

[0053] According to the semantic level, memory gate, storage gate and update gate are constructed respectively;

[0054] Constructing a context controller according to the memory gate, the storage gate, the first forget gate coefficient and the target multimodal feature;

[0055] Constructing a multimodal feature map according to the context controller, the second forget gate coefficient, the update gate and the target multimodal feature;

[0056] Constructing a probability map according to the preset number of semantic levels, the multimodal feature maps at different semantic levels, the first preset convolutional layer and the preset activation function;

[0057] Constructing a binary cross entropy loss function based on the probability map, the set of image pixels, and the segmentation mask of the segmentation gold standard;

[0058] Constructing a total loss function according to the binary cross entropy loss function, the target cosine loss function and the empirical hyperparameters;

[0059] Updating the multimodal feature map using the total loss function;

[0060] Update the probability map according to the updated plurality of multimodal feature maps;

[0061] The lung image segmentation result is generated according to the updated probability map.

[0062] On the other hand, an embodiment of the present invention provides a lung image segmentation device, comprising:

[0063] The first module is used to obtain lung images and radiology report information;

[0064] The second module is used to preprocess the lung image to obtain initial image spatial features and initial image channel features;

[0065] The third module is used to pre-process the radiology report information to obtain initial spatial level text features and initial channel level text features;

[0066] A fourth module is used to perform multimodal feature alignment processing using a target cosine loss function according to the initial image spatial features, the initial image channel features, the initial spatial level text features and the initial channel level text features to obtain target alignment features, wherein the target alignment features include target image spatial features, target image channel features, target spatial level text features and target channel level text features;

[0067] A fifth module is used to construct a spatial-level multimodal feature according to the target image spatial feature and the target spatial-level text feature;

[0068] A sixth module is used to construct a channel-level multimodal feature according to the target image channel feature and the target channel-level text feature;

[0069] A seventh module is used to construct a target multimodal feature according to the dual cross-modal self-attention mechanism, the spatial multimodal feature and the channel multimodal feature;

[0070] The eighth module is used to perform adaptive gated fusion processing on the target multimodal features at different semantic levels according to a preset number of semantic levels to obtain a lung image segmentation result.

[0071] In another aspect, an embodiment of the present invention provides a computer device, comprising:

[0072] at least one processor;

[0073] at least one memory for storing at least one program;

[0074] When the at least one program is executed by the at least one processor, the at least one processor implements the described method.

[0075] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0076] The beneficial effects of the present invention are as follows:

[0077] The present invention first obtains lung images and radiology report information, then preprocesses the lung images to obtain initial image spatial features and initial image channel features, and preprocesses the radiology report information to obtain initial spatial-level text features and initial channel-level text features, and then uses the target cosine loss function to perform multimodal feature alignment processing to obtain target alignment features, and constructs spatial-level multimodal features and channel-level multimodal features. Finally, according to the dual cross-modal self-attention mechanism, spatial-level multimodal features and channel-level multimodal features, target multimodal features are constructed, and according to the preset number of semantic levels, adaptive gated fusion processing is performed on the target multimodal features at different semantic levels. After obtaining the classification probability map at the pixel level of the lung image, the lung image segmentation result is generated, thereby realizing lung image segmentation, improving accuracy and applicability, and reducing computational complexity.

[0078] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0080] Figure 1 This is a flow chart of a lung image segmentation method according to an embodiment of the present invention;

[0081] Figure 2 A schematic diagram of a lung image according to an embodiment of the present invention;

[0082] Figure 3 A schematic diagram of radiology report information according to an embodiment of the present invention;

[0083] Figure 4 A schematic diagram of a pure image segmentation result according to an embodiment of the present invention;

[0084] Figure 5 A schematic diagram of a text-assisted image segmentation result according to an embodiment of the present invention;

[0085] Figure 6 A schematic diagram of a gold standard lung image segmentation result according to an embodiment of the present invention;

[0086] Figure 7 This is an overall flow chart of lung image segmentation using a dual multimodal self-attention network according to an embodiment of the present invention;

[0087] Figure 8 A schematic diagram of an implementation process of feature alignment and dual cross-modal self-attention in an embodiment of the present invention;

[0088] Fig. 9 This is a schematic diagram of the structure of a lung image segmentation device according to an embodiment of the present invention;

[0089] Fig.10 The figure is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0091] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0092] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0093] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0094] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0095] Multimodal Deep Learning Network: A deep learning model architecture whose input data is multimodal data (images and text).

[0096] Self-Attention Mechanism: It is a commonly used attention mechanism in deep learning. This mechanism allows the model to focus on the information of different spaces in the sequence when processing sequence data, and assign different attention weights at different time steps. The basic idea is to calculate the degree of association between each space in the sequence and other spaces, and then assign a weight to each space based on these degrees of association. These weights are used to calculate the weighted average of each space to obtain the overall context representation. This mechanism enables the model to capture long-distance dependencies in the sequence more flexibly without being restricted by deep learning networks.

[0097] Cross-modal self-attention mechanism: Utilize the fusion of two modalities to form cross-modal features, and use the self-attention mechanism based on these cross-modal features.

[0098] Dual cross-modal self-attention mechanism: perform cross-modal self-attention mechanism at the spatial level and the channel level.

[0099] Lung target region segmentation: Lung target region segmentation is an important task in the field of medical image processing, which aims to accurately extract and mark the lung region of interest from medical images.

[0100] Convolutional Neural Networks (CNN): A type of feedforward neural network that includes convolutional calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representational learning capabilities and can classify input information in a translation-invariant manner according to their hierarchical structure, so they are also called translation-invariant artificial neural networks.

[0101] In the related art, text-assisted lung image segmentation is to combine the information of natural language text with medical images to improve the accuracy and interpretability of lung image segmentation. This method combines the techniques of natural language processing (NLP) and medical image processing, which helps to more comprehensively understand the structure and lesions of lung images. Text-assisted lung image segmentation includes the following technical features: (1) The need for information fusion: Lung images themselves provide rich information about lung structure, but often lack intuitive semantic labels. Combining text information can provide a more semantic interpretation of the image, thereby enhancing the segmentation algorithm's understanding of the image content. (2) Utilization of medical reports: Clinical medical reports usually contain detailed descriptions of lung images, including information such as the space, type, and size of the lesions. Combining this text information with image data can provide valuable prior knowledge for the segmentation algorithm. (3) Natural language processing technology: Using natural language processing technology, key information about lung lesions can be extracted from medical reports. This includes named entity recognition (NER) for identifying lesion names and relationship extraction for obtaining associations between lesions. (4) Multimodal information fusion: Text-assisted lung image segmentation usually also involves the fusion of multimodal information, such as combining text descriptions, CT scan images, and other clinical data. Such fusion can provide more comprehensive and multi-angle information, thereby enhancing the performance of the segmentation algorithm. (5) Deep learning and attention mechanism: Deep learning methods are widely used in text-assisted lung image segmentation. Deep learning architectures such as convolutional neural networks (CNNs) can be used to process image data, while attention mechanisms can enable the model to pay more attention to key image areas, thereby improving segmentation accuracy. (6) Joint modeling of images and text: Some methods use a joint modeling approach to process image and text information at the same time. This can be achieved through shared embedding spaces, joint loss functions, etc., so that the model can fully utilize the complementarity of the two types of information. (7) Interpretability and clinical application: One of the advantages of text-assisted lung image segmentation is that it improves the interpretability of the model. By associating text descriptions with image segmentation results, it is easier to understand the model's decisions, thereby promoting clinical application. The development of text-assisted lung image segmentation is expected to provide more comprehensive and accurate solutions for medical image analysis. Traditional methods, given an input image and a natural language expression, calculate the object referred to by the segmented language expression in the image. Traditional methods use text descriptions to accurately segment the target referred to by the reference image. By introducing a cross-modal self-attention mechanism, the network can effectively capture the long-range dependencies between language and visual features, and can adaptively focus on the informative words in the reference expression and the important areas in the input image.However, the disadvantages of the existing technology include: (1) The model has a high computational complexity, which leads to increased computational costs. For example, the image features on each image pixel are repeatedly connected to the text features of each word to form cross-modal features, resulting in a waste of computing power and implementation redundancy. (2) The model is limited to long-distance associations. Due to excessive computing power consumption and high computational complexity, the model limits the text input length to less than 20 words, which is inconsistent with the real-world text situation and is not conducive to the application of the model in the real world, especially in the medical field. For example, case text cannot be restricted by the length of words and has low applicability. (3) The performance of the model is highly dependent on the quality and quantity of training data. When the training data is insufficient, such as for small sample data sets, the model performs poorly and the generalization performance is insufficient. (4) The cross-modal self-attention of the model only models at the global feature level and ignores the local feature modeling. For example, only the image features at each pixel position are combined with the text to establish cross-modal features, resulting in the model not being able to obtain the optimal solution. (5) Before the model forms cross-modal features, it does not perform feature alignment, which will lead to inconsistent feature spaces of features of different modalities. The formed cross-modal features cannot truly inherit the features of the two modalities, affecting the segmentation effect of subsequent cross-modal feature fusion and resulting in low segmentation accuracy.

[0102] Lung target region segmentation is a key task in the field of medical image processing. Its goal is to accurately extract and segment the lung region of interest from chest medical images. These medical images are generally from X-rays or CT scans, which can provide detailed information about lung structure and lesions. For lung target regions in actual application scenarios, such as lung nodules, atelectasis and other lung lesion areas, foreign objects in the lungs, etc., image segmentation algorithms are based on deep learning technology, especially convolutional neural networks (CNNs). Compared with traditional image processing technology, they can learn more complex features in the image, making the segmentation of lung target regions more accurate. In order to evaluate the performance of the segmentation algorithm, evaluation indicators such as the Dice coefficient and mean intersection over union (IOU) are usually used to compare the consistency between the segmentation results and the gold standard manually labeled by professional doctors. However, a simple CNN image segmentation algorithm sometimes cannot correctly segment the target area in the image, and requires segmenting the lung area from the input image, that is, first identifying the complete lung area, and then removing the atelectasis area to obtain the remaining lung area. After the CNN segmentation algorithm, compared with the gold standard, the simple image segmentation result obviously failed to identify the atelectasis (incomplete air content) of the right upper lung, and part of it was misjudged as the lung expansion area. It can be concluded that a single image modality cannot express the complete and true semantic features. After introducing the text description of the doctor's impression, the text input clearly mentioned that the right upper lung was incompletely aired. The segmentation accuracy was improved by using the multimodal deep learning segmentation algorithm of the text-assisted image. It can be seen that the atelectasis area of ​​the right upper lung has been correctly identified and eliminated from the lung expansion area. It can be concluded that the multimodal features formed by introducing text features and finding correlations with image features can better characterize the category attributes of the image pixel level, which can improve the accuracy of segmenting the lung expansion area.

[0103] In view of this, the present invention first extracts features from both image and text modalities and aligns feature spaces, then constructs two dual cross-modal self-attention modules that can capture long-range contextual correlations between images and texts in parallel at the image space level and channel level, and finally uses a gating mechanism to adaptively fuse cross-modal joint features from different semantic levels to generate image features assisted by text representation for accurate segmentation of chest X-ray target areas. Among them, the dual cross-modal self-attention network (Dual Multi-Modal Self-Attention Network, DMMSA Net) proposed in the present invention, given a chest X-ray and the corresponding radiology report as auxiliary information of the image, first uses a CNN network to extract visual feature maps at different semantic levels for the input image. Each word in the radiology report is represented as a word embedding vector. After the feature spaces of each modality are aligned, each word vector is then combined with the image space level features and the channel level features to form cross-modal joint features at two levels, thereby improving the segmentation accuracy.

[0104] The overall framework of the embodiment of the present invention is as follows: First, in order to solve the problem of noise in the image feature expression of accurate lung target area segmentation, it is necessary to make the auxiliary image feature expression of the lesion area words (such as upper left, upper right, etc.) of the text better. However, its potential limitation is that when LSTM captures the vector generated by the text feature expression for the entire text, it lacks the full expression of the detailed information of some individual words, and these words are precisely crucial to the better auxiliary image feature expression. The embodiment of the present invention can solve this problem and does not use LSTM to encode the overall text expression. It can better capture more detailed information at the word level. At the same time, due to the different initial feature space levels of word features and image features, the heterogeneity and non-uniform feature expression of multimodal features bring difficulties to the later fusion. Therefore, this embodiment uses the target cosine loss function to approach the consistency of the two modal feature spaces as much as possible. Second, traditional methods process each word in the text and then splice it with image features to infer the target area, which limits these methods to focus only on local spatial regions, but lacks the correlation of long-distance spatial regions in the global context, which is the key to understanding and segmenting tasks at the semantic level, and it is useless to express image features more comprehensively at both the global and local levels. Therefore, this embodiment uses a dual cross-modal self-attention module to effectively simulate the long-range dependencies between language and image modalities not only from the local (channel level) features of the image, but also from the global (spatial level) features of the image. In addition, cross-modal self-attention can better capture the long-range dependencies between visual and language contexts than inter-modal cross-attention. Compared with cross-attention, the main advantage of cross-modal self-attention is its greater flexibility and internal association ability, which can naturally integrate and process multimodal data without explicitly designing cross-modules. Third, this embodiment uses an adaptively selective gated fusion module to combine self-attention features at different semantic levels, and the feature fusion accuracy is higher.

[0105] A lung image segmentation method provided in an embodiment of the present application relates to the field of image processing technology. A lung image segmentation method provided in an embodiment of the present application can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in a blockchain network; the software can be an application that implements a lung image segmentation method, etc., but is not limited to the above forms.

[0106] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0107] The following is a detailed explanation of the embodiments of the present application in conjunction with the accompanying drawings:

[0108] Figure 1 is an optional flowchart of a lung image segmentation method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S108.

[0109] Step S101, obtaining lung images and radiology report information;

[0110] Step S102, preprocessing the lung image to obtain initial image spatial features and initial image channel features;

[0111] Step S103, preprocessing the radiology report information to obtain initial spatial level text features and initial channel level text features;

[0112] Step S104: performing multimodal feature alignment processing using a target cosine loss function according to the initial image spatial features, the initial image channel features, the initial spatial level text features, and the initial channel level text features to obtain target alignment features, where the target alignment features include target image spatial features, target image channel features, target spatial level text features, and target channel level text features;

[0113] Step S105, constructing a spatial-level multimodal feature according to the target image spatial feature and the target spatial-level text feature;

[0114] Step S106: construct channel-level multimodal features according to the target image channel features and the target channel-level text features;

[0115] Step S107: constructing target multimodal features according to the dual cross-modal self-attention mechanism, the spatial multimodal features and the channel multimodal features;

[0116] Step S108: According to a preset number of semantic levels, adaptive gating fusion processing is performed on the target multimodal features at different semantic levels to obtain a lung image segmentation result.

[0117] Steps S101 to S108 shown in the embodiment of the present application implement lung image segmentation, improve accuracy and applicability, and reduce computational complexity.

[0118] In step S101 of some embodiments, the lung image and radiology report information may be obtained through a medical information database, or may be obtained through other methods, not limited thereto.

[0119] In some embodiments, in step S102, the lung image is preprocessed to obtain initial image spatial features and initial image channel features, which may include but is not limited to the following steps:

[0120] Extracting an initial feature map from a lung image;

[0121] Extract visual features from the initial feature map;

[0122] Perform L2 normalization on visual features;

[0123] According to the feature map size, pixel coordinates and preset spatial coordinate feature dimensions, construct the spatial coordinate features corresponding to each pixel in the initial feature map, the spatial coordinate features including horizontal position information of the first vector dimension, vertical position information of the second vector dimension, image width information of the third vector dimension and image height information of the fourth vector dimension;

[0124] Perform L2 normalization on the spatial coordinate features;

[0125] Constructing initial image spatial features based on the spatial coordinate features normalized by the L2 paradigm and the visual features normalized by the L2 paradigm;

[0126] The visual features normalized by L2 paradigm are transformed to obtain the initial image channel features.

[0127] In some embodiments, a base CNN neural network may be used to extract an initial feature map from a lung image. The feature map extracted from a specific CNN layer is represented as Among them, H represents the height of the feature map, W represents the width of the feature map, and C i Represents the feature map channel value. For the convenience of expression, this embodiment only uses the features extracted from a specific CNN layer for example. It is further fully demonstrated in the subsequent adaptive gated fusion, which includes features from different CNN layers. Then, visual features are extracted from the initial feature map, and a spatial global feature expression can be constructed. At each spatial position, set P to be a pixel position on the feature map I, for example, p∈{1,2,…,H×W}. Using To define the visual features of the spatial position P in the image, where C i is the number of channels of the feature map of this layer. p Perform L2 normalization, that is Among them, L2 paradigm normalization is a paradigm normalization method for machine learning. In addition to image features, spatial coordinate features have also been proven to be useful for semantic segmentation tasks. The spatial coordinate features corresponding to each pixel in the initial feature map can be constructed based on the feature map size, pixel coordinates and preset spatial coordinate feature dimensions, wherein the spatial coordinate features include horizontal position information of the first vector dimension, vertical position information of the second vector dimension, image width information of the third vector dimension and image height information of the fourth vector dimension. Exemplarily, the preset spatial coordinate feature dimension can be set to 8, the first vector dimension can be set to 3, the second vector dimension can be set to 3, the third vector dimension can be set to 1, and the fourth vector dimension can be set to 1, and an 8-dimensional spatial coordinate vector is defined for each pixel position, wherein 3 dimensions encode horizontal position information, 3 dimensions encode vertical position information, 1 dimension encodes image width information, and 1 dimension encodes image height information. Among them, the spatial coordinate feature of position P is defined as Then, the spatial coordinate features are normalized by L2 paradigm, and then the initial image spatial features are constructed based on the spatial coordinate features normalized by L2 paradigm and the visual features normalized by L2 paradigm. The calculation formula of the initial image spatial features is: In the formula, v p is the initial image spatial feature, Concat(·) is the link between the image spatial feature and the position feature, ∥i p ∥2 is the L2 norm of the visual feature, is the L2 normalization of visual features, i p is the visual feature, s p is the spatial coordinate feature, and 0 represents the link in the feature dimension. It can be understood that the initial image spatial feature v p It encodes the global features of a specific position P on the image, and the vector dimension is (C i +8). ​​You can use To represent the image features at different spatial positions. The dimension of V is H×W×(C i +8). ​​Among them, is a discrete mathematical symbol that represents the set of all pixel points P on the feature map, then V is the image feature of point P at all different spatial positions on the map. Finally, the visual features normalized by the L2 norm are transformed to obtain the initial image channel features. We can set N = H × W and transform the feature map Reshape The image channel feature (image local feature) vector of a specific layer is 1×N. To represent the image features at different channel positions, the dimension of I is C i ×N. c can be set to be one of all feature maps in this layer, for example, c∈{1,2,…,Ci}.use To define the channel features of the cth feature map in this layer, N is the number of pixels in a single feature map of this layer. c by i p The feature transformation after L2 normalization is performed, so L2 normalization is no longer required; secondly, since the image features at different channel positions are not directly related to the pixel coordinate information, the spatial coordinate feature s is not required. p To form I. For example, when 1600 1x500 visual features i p After L2 normalization, all element values ​​of the visual features are already between [0, 1]. The element values ​​of the 500 1x1600 feature vectors are the normalized element values, but the form of the feature vectors has changed, the dimension has changed, and the number has changed, but the element values ​​are the same. Therefore, the 500 1x1600 initial image channel features i after the representation transformation are c It is no longer necessary to normalize. It is understandable that v p The visual feature i p Then add the 8-dimensional spatial features of each P point. In addition, this embodiment not only establishes the image spatial features (global image features), but also uses the feature map itself to establish the image channel features (local image features). Due to the underlying mechanism of the convolutional neural network, the feature map itself can capture the local features of objects on the image. The channel local features expressed by the feature map, combined with the spatial global features, can more comprehensively characterize the image. Finding the contextual relationship between these feature maps themselves and the text words is also conducive to better expressing the multimodal joint features passed through the attention module. At the same time, unlike other works that use random generation or sequential division of small blocks of images, and then use CNN to extract local features, the image channel features established in this embodiment do not increase the complexity of the network, and save additional computing power to divide and extract the features of the regional blocks on the image. Using only the feature map of the base CNN, while obtaining the spatial global features, the channel local feature expression is also obtained. In addition, cross-modal features are subsequently established for three different levels of image features. The feature maps at different levels express different semantic information, which can help the model better find the long-range correlations between cross-modal contexts at different semantic levels, more comprehensively characterize image features, and further fully characterize cross-modal features.

[0128] In some embodiments, in step S103, the radiology report information is preprocessed to obtain initial spatial level text features and initial channel level text features, which may include but is not limited to the following steps:

[0129] The radiology report information is processed by word division to obtain a word set;

[0130] Perform one-hot encoding on the word set to obtain multiple one-hot vectors;

[0131] Perform word embedding processing on the one-hot vector to obtain the word vector;

[0132] According to the image space feature dimension of the initial image space feature, a first linear transformation is performed on the word vector so that the feature dimension of the word vector after the first linear transformation is consistent with the image space feature dimension;

[0133] Perform L2 normalization on the word vector after the first linear transformation to obtain the initial spatial level text features;

[0134] According to the image channel feature dimension of the initial image channel feature, a second linear transformation is performed on the word vector, so that the feature dimension of the word vector after the second linear transformation is consistent with the image channel feature dimension;

[0135] The word vector after the second linear transformation is normalized using the L2 norm to obtain the initial channel-level text features.

[0136] In some embodiments, the radiology report information (i.e., text) can be first processed by word segmentation to obtain a word set. For example, for a text description containing K words, i.e., radiology report information, for a non-public Chinese data set, the word segmentation tool CMeKG developed based on Bert and CBlue can be used to divide the report into K words, while for a public English data set, no additional processing is required. Then, the word set is one-hot encoded, and each word is encoded as a one-hot vector to obtain multiple one-hot vectors. The one-hot vector is then word-embedded and mapped to a compact word embedding obtained by a query table, i.e., a word vector is obtained. Among them, the one-hot vector can be represented as In the formula, C t =1000, C t is the dimension of the word embedding feature space. It can be understood that, unlike the traditional method of using LSTM to sequentially process word vectors and encode the entire text description as a sentence vector, this embodiment retains each word vector and introduces a cross-modal self-attention module to capture the long-distance correlation between these words and spatial regions on the image. Then, according to the image space feature dimension of the initial image space feature, a first linear transformation is performed on the word vector so that the feature dimension of the word vector after the first linear transformation is consistent with the image space feature dimension, and the word vector after the first linear transformation is normalized by L2 norm to obtain the initial spatial level text feature. Among them, the linear transformation of features refers to the use of a 1x1 convolutional layer to increase or decrease the dimension of each type of feature on a specified dimension. Suppose the word vector t of the text after word embedding processing k To report the features of the kth word, the feature space level is C t . K is the number of words in the text. In the image space feature dimension, the word vector t k After the first linear transformation and L2 normalization, the initial spatial level text feature t is obtained. k Among them, the feature space level is (C i +8), which is consistent with the spatial level of the image spatial features. Finally, according to the image channel feature dimension of the initial image channel feature, the word vector is subjected to a second linear transformation, so that the feature dimension of the word vector after the second linear transformation is consistent with the image channel feature dimension, and the word vector after the second linear transformation is normalized by L2 norm to obtain the initial channel level text feature. In the image channel feature dimension, the word vector t k Perform the second linear transformation and L2 normalization to obtain the initial channel-level text features t k ′, the feature space level is N, which is consistent with the spatial level of the image channel features. K is the number of text words. It can be understood that in the same feature space, the text word features are respectively aligned with the image features in two dimensions. So that the alignment of the two modal features can be approximated by maximizing the cosine similarity between vectors during model training and fitting. In linear transformation, the parameters are trainable, and the parameters of the first linear change and the second linear transformation are different. The general formula can be Y=WX+b, where W and b are both trainable linear transformation weights, X is the input, and Y is the linear transformation output for X. More importantly, this embodiment linearly transforms the text word vectors respectively, and maps them to the feature space of the global features of the image space and the local features of the image channel, so that the image and text modes can be aligned in parallel in the image space feature and image channel feature dimensions. And choose to use cosine similarity to measure the distance between the two modal features, and minimize the cosine loss function to approximate the features of the two modalities.

[0137] In some embodiments, in step S104, a multimodal feature alignment process is performed using a target cosine loss function according to the initial image spatial features, the initial image channel features, the initial spatial level text features, and the initial channel level text features to obtain a target alignment feature, which may include but is not limited to the following steps:

[0138] According to the initial image spatial features and the number of pixels in the feature map, the image spatial average features are constructed;

[0139] Construct the image channel average feature according to the initial image channel feature and the number of feature map channels;

[0140] According to the initial spatial level text features and the number of text words, the spatial level text average features are constructed;

[0141] Construct channel-level text average features based on the initial channel-level text features and the number of text words;

[0142] According to the spatial average features of the image and the spatial average features of the text, a spatial cosine loss function is constructed;

[0143] Construct a channel-level cosine loss function based on the image channel average features and the channel-level text average features;

[0144] Construct the target cosine loss function according to the spatial level cosine loss function and the channel level cosine loss function;

[0145] The target cosine loss function is used to update the initial image spatial features, initial image channel features, initial spatial level text features and initial channel level text features to obtain the target alignment features.

[0146] In some embodiments, the cosine loss function can be used in two feature dimensions to align the text word features and image features at the spatial and channel levels of the image, respectively, to avoid the magnitude difference between the two feature vectors being too large, which may lead to deviations in subsequent attention weight scores. The image space average feature can be constructed based on the initial image space feature and the number of pixels in the feature map, where the expression of the image space average feature is: In the formula, is the image space average feature, N is the number of pixels in the feature map, v p is the initial image spatial feature. According to the initial image channel feature and the number of feature map channels, the image channel average feature is constructed, where the expression of the image channel average feature is: In the formula, is the average feature of the image channel, C i is the number of feature map channels, i c is the initial image channel feature. It can be understood that v p is 1×(C i +8) features, there are N = H × W in total, H is the feature map height, W is the feature map width; i c It is a 1×N feature, with a total of C i According to the initial spatial level text features and the number of text words, the spatial level text average features are constructed, where the expression of the spatial level text average features is: In the formula, is the average feature of the text at the spatial level, K is the number of text words, and t k is the initial spatial level text feature. According to the initial channel level text feature and the number of text words, the channel level text average feature is constructed, where the expression of the channel level text average feature is: In the formula, is the average feature of text at the channel level, t′ k is the initial channel-level text feature. Based on the image spatial average feature and the spatial-level text average feature, a spatial-level cosine loss function is constructed, where the expression of the spatial-level cosine loss function is: Where, L cos_p is the spatial cosine loss function, and ∥.∥ is the L2 normalization process. According to the average features of the image channels and the average features of the channel-level text, the channel-level cosine loss function is constructed, where the expression of the channel-level cosine loss function is: Where, L cos_c is the channel-level cosine loss function. According to the spatial-level cosine loss function and the channel-level cosine loss function, the target cosine loss function is constructed, where the expression of the target cosine loss function is: cos =avg(L cos_p ,L cos_c ), where L cos is the target cosine loss function. Then, the target cosine loss function is used to update the initial image space features, the initial image channel features, the initial space-level text features, and the initial channel-level text features to obtain the target alignment features. Among them, the target alignment features may include target image space features, target image channel features, target space-level text features, and target channel-level text features. Furthermore, the target cosine loss function can be used as one of the loss losses in multi-loss joint learning (Joint Learning) in the subsequent process. The features of each modality output by the feature space alignment module in this embodiment can be used as the input of the subsequent cross-modal self-attention module.

[0147] In some embodiments, in step S105, constructing a spatial-level multimodal feature based on the target image spatial feature and the target spatial-level text feature may include but is not limited to the following steps:

[0148] Combine the target image spatial features and the target spatial level text features to obtain the spatial level cross-modal joint features;

[0149] According to the cross-modal joint features at the spatial level, the query vector, key vector and value vector are constructed respectively;

[0150] Construct an attention score based on the query vector and the key vector;

[0151] Construct a value weighted vector based on the attention score and the value vector;

[0152] According to the linear transformation weight, a third linear transformation is performed on the value weighted vector so that the feature dimension of the value weighted vector is consistent with the feature dimension of the cross-modal joint feature at the spatial level;

[0153] According to the value weighted vector after the third linear transformation and the spatial cross-modal joint feature, a spatial cross-modal self-attention feature is constructed;

[0154] Based on the cross-modal self-attention features at the spatial level, multimodal features at the spatial level are constructed. The channel dimension of the multimodal features at the spatial level after the fourth linear transformation is 500 dimensions.

[0155] In some embodiments, the target image spatial feature and the target spatial level text feature may be combined to obtain a spatial level cross-modal joint feature, wherein the expression of the spatial level cross-modal joint feature is: pk =Concat(V,T,1), where F pk is the spatial cross-modal joint feature, Concat(.) is the feature combination processing, V is the feature dimension of the target image spatial feature, T is the feature dimension of the target spatial text feature, and 1 is the link in the number of features. It can be understood that the spatial cross-modal joint feature F pk It encodes the combination of image spatial features (image position features) and text word features, where the spatial level cross-modal joint feature F pk The dimensions are (H×W+K)×(C i +8). ​​In this embodiment, the number of words in the text may not be limited, and the spatial cross-modal joint feature F pk The feature dimension of is fixed and does not change with the length of the text words in the radiology report. And by using the parallel mechanism of deep learning, the computing power occupied is greatly reduced. In addition, the attention mechanism can capture important information from the rough original text or visual features. This embodiment uses the attention of multimodal features, especially the dual multimodal self-attention module obtained by improving the self-attention mechanism, which can capture the long-range dependency correlation between words in the radiology report and different areas on the X-ray chest film. The spatial cross-modal joint feature F can be used later. pk and channel-level cross-modal joint features F ck As input, after learning the correlation between the text description and the image context, a new feature map is generated by summarizing, where the size of the feature map is independent of the number of words in the text description. Then, at the image spatial feature level, the cross-modal joint feature F pk , through linear transformation, construct query vector, key vector and value vector respectively. Among them, the expression of query vector is: Q pk =W q F pk , where Q pk is the query vector, W q is the first model parameter. The key vector is expressed as: K pk =Wk F pk , where K pk is the key vector, W k is the second model parameter. The expression of the value vector is: pk =W v F pk , where V pk is a value vector, W v is the third model parameter. For each spatial position P and K-th word on the image, the first model parameter, the second model parameter and the third model parameter can be obtained through training. Exemplarily, in order to improve the computational efficiency, each query vector, key vector and value vector is reduced from the high-dimensionality of the multimodal feature to 512 dimensions, such as Then, based on the query vector and the key vector, an attention score is constructed, where the calculation formula of the attention score is: pk =Softmax(Q pk T ,K pk ), in the formula, sa pk is the attention score, which can also be called the self-attention similarity matrix of image and text cross-modality at the spatial level, quantifying the correlation between each image spatial position P and each text description word K. Then, based on the attention score and the value vector, a value weighted vector is constructed, where the calculation formula of the value weighted vector is: In the formula, is the value-weighted vector, p is the total number of spatial positions on the image, and k is the total number of words. Then, according to the linear transformation weight, the value-weighted vector is subjected to a third linear transformation, so that the feature dimension of the value-weighted vector is consistent with the feature dimension of the cross-modal joint feature at the spatial level, so that the value-weighted vector and the cross-modal self-attention feature at the spatial level are subsequently added, and their feature dimensions are consistent. Then, according to the value-weighted vector after the third linear transformation and the cross-modal joint feature at the spatial level, the cross-modal self-attention feature at the spatial level is constructed, where the calculation formula of the cross-modal self-attention feature at the spatial level is: In the formula, is the cross-modal self-attention feature at the spatial level, is the linear transformation weight. Among them, Finally, based on the spatial cross-modal self-attention features, the spatial multimodal features are constructed, where the channel dimension of the spatial multimodal features after the fourth linear transformation is 500. The final representation form of the spatial multimodal features is the feature map, so it has the number of channels, that is, the dimension of the channel direction. Exemplarily, the spatial cross-modal self-attention features can be divided into feature dimensions of (H×W)×(C i +8) image features and feature dimensions are K×(C i+8) text word features, so the K text word features can be averaged and then concatenated to the above image space features to form a spatial multimodal feature, and the channel dimension of this spatial multimodal feature is converted to 500 dimensions. Define a multimodal feature that fuses text and images at all pixel locations is a collection of multimodal features at the spatial level.

[0156] In some embodiments, in step S106, constructing a channel-level multimodal feature based on the target image channel feature and the target channel-level text feature may include but is not limited to the following steps:

[0157] Combine the target image channel features and the target channel-level text features to obtain the channel-level cross-modal joint features;

[0158] Construct an attention map based on the cross-modal joint features at the channel level;

[0159] Based on the attention map and the cross-modal joint features at the channel level, the channel-level cross-modal self-attention features are constructed;

[0160] Based on the cross-modal self-attention features at the channel level, channel-level multimodal features are constructed. The channel dimension of the channel-level multimodal features after the fifth linear transformation is 500 dimensions.

[0161] In some embodiments, the target image channel feature and the target channel-level text feature can be first combined to obtain a channel-level cross-modal joint feature, wherein the expression of the channel-level cross-modal joint feature is: ck =Concat(I,T′,1), where F ck is the channel-level cross-modal joint feature, Concat(.) is the feature combination processing, I is the feature dimension of the target image channel feature, T′ is the feature dimension of the target channel-level text feature, and 1 is the link in the number of features. It can be understood that the channel-level cross-modal joint feature F ck It encodes the combination of image channel features and text word features, where the channel-level cross-modal joint feature F ck The dimension is (C i +K)×N. In this embodiment, the number of words in the text may not be limited, and the channel-level cross-modal joint feature F ck The feature dimension of is fixed and does not change with the length of the text words in the radiology report. Then, the attention map is constructed based on the cross-modal joint features at the channel level. At the image channel feature level, given a multimodal feature F ck , which is different from the self-attention implementation at the spatial feature level, but can directly calculate the multimodal features at the channel feature level Attention map Among them, the calculation formula of the attention map is: ck =Softmax(F ck T , F ck ), in the formula, sa ck is the attention map. It is understandable that F can be reshaped ck for Where N = H × W, then realize F ck and its transpose F ck T Finally, a Softmax layer is applied to obtain the channel-level attention map The relationship between each channel map c and each word k of the image is obtained. Then, based on the attention map and the cross-modal joint features at the channel level, the cross-modal self-attention features at the channel level are constructed. The calculation formula of the cross-modal self-attention features at the channel level is: In the formula, is the cross-modal self-attention feature at the channel level, c is the total number of channels on the image, and k is the total number of words. Finally, based on the channel-level cross-modal self-attention features, channel-level multimodal features are constructed, where the channel dimension of the channel-level multimodal features after the fifth linear transformation is 500. The final representation form of the channel-level multimodal features is the feature map, so it has the number of channels, that is, the dimension in the channel direction. The number of feature maps at the channel level is also called the number of channels. For example, a 40x40x500 feature map has a total of 500 40x40 feature maps, and the number of channels is 500. Similarly, the channel-level cross-modal self-attention features can be divided into feature dimensions of C i ×N image features and K×N text word features, so the K text word features can be averaged and then concatenated to the image space features to form a channel-level multimodal feature, and the channel dimension of this channel-level multimodal feature is converted to 500 dimensions. Multimodal features that combine text word features and image channel features are defined at all channel positions in the feature map is a collection of channel-level multimodal features.

[0162] In some embodiments, in step S107, a target multimodal feature may be constructed based on the dual cross-modal self-attention mechanism, the spatial multimodal feature, and the channel multimodal feature. The calculation formula of the target multimodal feature is: Where F is the target multimodal feature, is the multimodal feature at the spatial level, is the multimodal feature at the channel level. It can be understood that the image space level and the channel level and By summing up, we can get the final target multimodal feature F at a specific semantic level, whose dimension is 500. It can be understood that the dual cross-modal self-attention mechanism can perform cross-modal self-attention mechanisms at the spatial level and the channel level.

[0163] In some embodiments, in step S108, adaptive gating fusion processing is performed on target multimodal features at different semantic levels according to a preset number of semantic levels to obtain a lung image segmentation result, which may include but is not limited to the following steps:

[0164] According to the semantic level, memory gate, storage gate and update gate are constructed respectively;

[0165] Construct a context controller according to the memory gate, storage gate, first forget gate coefficient and target multimodal features;

[0166] Constructing a multimodal feature map according to the context controller, the second forget gate coefficient, the update gate and the target multimodal features;

[0167] Constructing a probability map according to a preset number of semantic levels, multimodal feature maps at different semantic levels, a first preset convolutional layer, and a preset activation function;

[0168] Construct a binary cross entropy loss function based on the probability map, the image pixel set, and the segmentation mask of the segmentation gold standard;

[0169] Construct the total loss function based on the binary cross entropy loss function, the target cosine loss function and the empirical hyperparameters;

[0170] Use the total loss function to update the multimodal feature map;

[0171] Update the probability map according to the updated multiple multimodal feature maps;

[0172] According to the updated probability map, a lung image segmentation result is generated.

[0173] In some embodiments, after obtaining the output F of a certain semantic level layer of CNN, fusing features of different scales can improve the effect of semantic segmentation. Memory gates, storage gates, and update gates can be constructed according to the semantic level. For example, for the i-th semantic level, a memory gate m is generated. i , Storage Doors i and update gate r i. The memory gate, storage gate and update gate are similar to the control gates in the long short-term memory network (LSTM). Different from the stage memory update, the calculation of the gates at each semantic level in this embodiment is decoupled from other semantic levels. The gates at the current semantic level together control how much the multimodal features at the current semantic level ultimately contribute to the final fusion features. Then, a context controller is constructed based on the memory gate, storage gate, first forget gate coefficient and target multimodal features. Each semantic level has a context controller G i To simulate the information flow from other semantic levels to the i-th semantic level. The calculation formula of the context controller is: G k =∑ q∈{1,2,3}|{k} α q (1-m q )⊙F q +s k ⊙F k , where G k is the context controller at the kth semantic level, α q is the first forget gate coefficient of the qth semantic level, m q is the memory gate of the qth semantic level, s k is the storage gate of the kth semantic level, F q is the target multimodal feature at the qth semantic level, F k is the target multimodal feature of the kth semantic level, and ⊙ represents the Hadamard matrix product, that is, the dot product of the matrix. It can be understood that the current semantic level is represented as k, and the other semantic levels are represented as q. Then, according to the context controller, the second forget gate coefficient, the update gate and the target multimodal feature, a multimodal feature map is constructed, where the calculation formula of the multimodal feature map is: In the formula, is the multimodal feature map of the kth semantic level, β is the second forget gate coefficient, r k is the update gate of the kth semantic level. Then, a probability map is constructed according to the preset number of semantic levels, the multimodal feature maps at different semantic levels, the first preset convolutional layer and the preset activation function. It can be understood that after the multimodal feature maps at three semantic levels are obtained by gated fusion and added and summed, a 3x3 convolutional layer and a nonlinear activation function are applied to obtain the probability map. The preset activation function may include a sigmoid activation function. In order to obtain the segmentation mask, the multimodal feature maps F from the three semantic levels may be superimposed. o (i) , i∈{1,2,3}, and use a 3×3 convolutional layer and sigmoid activation function. This sequence of operations outputs a probability map to represent the probability of each pixel being classified as foreground in the segmentation mask. The probability map is calculated as: Where P t is a probability map, σ(·) is the sigmoid activation function, is a 3×3 convolution operation. Then, based on the probability map, the image pixel set, and the segmentation mask of the segmentation gold standard, a binary cross entropy loss function is constructed, where the expression of the binary cross entropy loss function is: Where, L r is the binary cross entropy loss function, Ω is the set of image pixels, Y is the segmentation mask of the gold standard, n represents the nth pixel on the image, P t is a probability graph. Then, based on the binary cross entropy loss function, the target cosine loss function and the empirical hyperparameters, the total loss function is constructed. The calculation formula of the total loss function is: L total =λ*L cos +L r , where L total is the total loss function, λ is the empirical hyperparameter, L cos is the target cosine loss function, L r is the binary cross entropy loss function. The optimal value of the empirical hyperparameter can be found when debugging on the validation set, and the Adam algorithm can be used to optimize the total loss function. The multimodal feature map is then updated using the total loss function, and the probability map is updated based on the updated multiple multimodal feature maps. Finally, the lung image segmentation result is generated based on the updated probability map. It can be understood that this embodiment uses the ResNet-based DeepLab-101 as the base CNN, and the multimodal feature maps at three semantic levels (i=1,2,3) correspond to ResNet blocks Res3, Res4 and Res5. Let C vi is the number of channels of the multimodal feature map at the i-th semantic level of the network. (spatial cross-modal self-attention features at the i-th semantic level) is used to represent the cross-modal features of the image spatial features and the text word features after the dual multimodal self-attention module. is a multimodal feature for different spatial positions at the i-th semantic level. is at the i-th semantic level At the same time, use (Channel-level cross-modal self-attention features at the i-th semantic level) is used to represent the cross-modal features of the image channel features and the text word features after the dual multimodal self-attention module. It is a multimodal feature for different channel positions at the i-th level. Then, the multimodal features at the two levels of image spatial features (global image features) and image channel features (local image features) are converted to the same dimension, and then summed up to generate a multimodal feature map at the semantic level, so that the multimodal feature maps F at different levels can be fused. (i) (i=1,2,3) to generate a fusion feature map, and then generate the final lung image segmentation result. For example, the lung image obtained is as follows Figure 2 As shown in the figure, the radiology report information obtained is as follows Figure 3 When using a simple image segmentation method, the pure image segmentation result is as follows Figure 4 As shown; when using the traditional text-assisted image segmentation method, the text-assisted image segmentation results are as follows Figure 5 When the lung image segmentation method provided in this embodiment is used, the gold standard lung image segmentation result is as shown in Figure 6 As shown, it can be seen that the segmentation accuracy of the method in this embodiment is higher.

[0174] In some embodiments, the overall process of lung image segmentation using dual multimodal self-attention network is as follows: Figure 7As shown in the figure, the DMMSA (Dual Multi-modal Self Attention) network can be used for image segmentation. First, the CNN convolutional neural network Deeplab-Resnet01 is used as the backbone network to obtain the initial features of the image output at the three semantic levels of Res3, Res4 and Res5 from low to high semantic levels, and then the CMeKG word segmentation tool based on Chinese Bert and the one-hot encoding query table are used to obtain the word vector of the text as the initial text feature. The features of the two modalities are input into the feature alignment module. After approximating the feature space of the two modalities by calculating the cosine similarity between the vectors and using the cosine loss function, the aligned features of each modality with the same feature space are output. These features are then input into the dual cross-modal self-attention mechanism module. The cross-modal self-attention mechanism is performed on the image features and text word features at the two levels of spatial features and channel features of the image, and the attention weights of all features generated by the cross-modal self-attention mechanism are obtained to weightedly represent each feature. Then, the image and text features that have passed the self-attention mechanism are connected in series at the pixel level to capture the long-distance contextual correlation between the image and the text from the global (spatial features) and local (channel features). Here, the multimodal features obtained from the two dimensions are weighted and summed to have better semantic features. Finally, the multimodal features of each semantic level (Res3, Res4, Res 5) with rich semantic features are gated multi-level fused based on the gate mechanism to form the final pixel-level multimodal features. After passing through the pixel-level classification layer and using the cross-entropy loss function, the classification result at each pixel position is obtained, that is, the output image segmentation result. More, the specific process of feature alignment and dual cross-modal self-attention is as follows: Figure 8As shown in the figure, for the image feature input layers of res3 to res5 of deeplab-resnet101, feature alignment and dual cross-modal self-attention are performed at each level, and then the multimodal features formed at these three semantic levels are integrated to obtain the pixel-level classification results (i.e., lung image segmentation results). In the feature alignment module, it is assumed that Res5 outputs an image feature block of H*W*C, where H is the image height, W is the image width, and C is the number of feature map channels. After coupling the text word features into the feature alignment module, the text word vectors are mapped to C dimensions and H*W dimensions at the spatial level and channel level, respectively, in order to align the features with the image features of these two dimensions. When passing through the dual cross-modal self-attention mechanism module, the text word features are parallel to the image features, and the cross-modal matrices C*(H*W+N) and H*W*(C+N) are input from the spatial (global) level and the channel (local) level, and the cross-modal self-attention matrices (H*W+N)*(HW+N) and (C+N)*(C+N) are generated respectively, and each feature represented by all feature weights and characteristics is obtained, in order to capture the long-range contextual correlation between image spatial features (image position features) and text features, as well as the semantic correlation between image channel feature maps and text features, and finally the multimodal features obtained in these two dimensions are weighted and summed. In some embodiments, the method of this embodiment can be used to conduct experiments, with a total of 1372 non-public lung area segmentation data sets, in which the training set, validation set, and test set are randomly divided in a ratio of 8:1:1. The experimental results show that the average Dice coefficient on the test set is 96.43%, which is 2.67% higher than the unimodal segmentation algorithm that only uses image input, and the average intersection-over-union ratio is 87.58%, which is 1.03% higher than the unimodal segmentation algorithm that only uses image input.

[0175] In some embodiments, the technical features of this embodiment include: (1) A DMMSA (Dual Multi-modal Self Attention) deep learning network is proposed, which is composed of three modules, namely feature alignment, cross-modal dual self-attention mechanism and multi-level multi-modal feature fusion, which are sequentially linked. Among them, the feature alignment module is responsible for approximating the feature space of each modality, and the input is the initial image features extracted by CNN and the initial text features output by the word embedding algorithm; the dual cross-modal self-attention mechanism module is responsible for finding the optimal expression of image and text features, and performing cross-modal self-attention operations with text features in the global and local dimensions of the image, respectively, to find the correlation of each modality feature, and express each feature by the weight sum of all features; the multi-level multi-modal feature fusion module is responsible for fusing the multi-modal features at three levels on a multi-level network with CNN as the backbone, and outputting the final multi-modal features for pixel-level classification. (2) The dual cross-modal self-attention mechanism of this embodiment innovatively places text word features in parallel with the global (spatial level) features and local (channel level) features of the image, instead of repeatedly adding them after the image features. This processing not only reduces computational complexity and cost, but also better expresses the features of each modality by finding the correlation between features of different modalities in both global and local dimensions. (3) This embodiment uses cosine similarity to calculate the distance between vectors of different modalities, and approximates the feature spaces of different modalities by using cosine loss. (4) In the design of multiple loss functions of this embodiment, the total loss function is the sum of the empirical weights of the three losses: feature alignment loss at the global / spatial level, feature alignment loss at the local / channel level, and pixel-level classification crossentropy loss, which ensures the convergence and stability of the model. (5) In the gated multi-level fusion design of this embodiment, the memory gate m is designed with reference to the control gate of LSTM. i and storage doors i and update gate r i ,Different from the stage memory update, the calculation of the gate at each level is decoupled from other levels. The gates at each level together control how much the multimodal features at each level ultimately contribute to the final fusion feature.

[0176] The beneficial effects of implementing the embodiments of the present invention include: the embodiments of the present invention first obtain lung images and radiology report information, then preprocess the lung images to obtain initial image spatial features and initial image channel features, and preprocess the radiology report information to obtain initial spatial level text features and initial channel level text features, and then use the target cosine loss function to perform multimodal feature alignment processing to obtain target alignment features, and construct spatial level multimodal features and channel level multimodal features, and finally construct target multimodal features according to the dual cross-modal self-attention mechanism, spatial level multimodal features and channel level multimodal features, and according to the preset semantic level number, adaptive gate fusion processing is performed on the target multimodal features at different semantic levels, and after obtaining the classification probability map at the pixel level of the lung image, the lung image segmentation result is generated, thereby realizing lung image segmentation, improving accuracy and applicability, and reducing computational complexity. At the same time, this embodiment does not repeatedly connect the text features of each word at each pixel position, but juxtaposes the image features and text word features at each pixel position to perform a self-attention mechanism, which greatly reduces the computational cost and reduces the computational complexity. This embodiment does not limit the length of the text input words. Since the word features and the image features at the pixel position are juxtaposed, text data containing more words can be processed, and the long-distance association is enhanced. This embodiment can perform well on small data sets, and it is a small data set in the medical field, which reduces the dependence on data quality and quantity. The text features of this embodiment not only perform a cross-modal self-attention mechanism with the global features of the image, but also perform a cross-modal self-attention mechanism with the local features of the image. Based on the sum of the self-attention weights from the local and global features of the image and the text words respectively, the image and text feature expressions can be better obtained. This embodiment aligns the features of the text and the image, so that the features of the two modes form an approximate feature space, and provides aligned features of each modality to the subsequent dual cross-modal self-attention module, which helps the attention module to find a better expression of each modality feature and improve the segmentation effect.

[0177] like Fig. 9 As shown, the embodiment of the present invention further provides a lung image segmentation device, comprising:

[0178] The first module 801 is used to obtain lung images and radiology report information;

[0179] The second module 802 is used to preprocess the lung image to obtain initial image spatial features and initial image channel features;

[0180] The third module 803 is used to pre-process the radiology report information to obtain initial spatial level text features and initial channel level text features;

[0181] The fourth module 804 is used to perform multimodal feature alignment processing using a target cosine loss function according to the initial image spatial features, the initial image channel features, the initial spatial level text features and the initial channel level text features to obtain target alignment features, where the target alignment features include target image spatial features, target image channel features, target spatial level text features and target channel level text features;

[0182] The fifth module 805 is used to construct a spatial-level multimodal feature according to the target image spatial feature and the target spatial-level text feature;

[0183] The sixth module 806 is used to construct a channel-level multimodal feature according to the target image channel feature and the target channel-level text feature;

[0184] The seventh module 807 is used to construct a target multimodal feature according to the dual cross-modal self-attention mechanism, the spatial multimodal feature and the channel multimodal feature;

[0185] The eighth module 808 is used to perform adaptive gated fusion processing on target multimodal features at different semantic levels according to a preset number of semantic levels to obtain a lung image segmentation result.

[0186] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0187] like Fig.10 As shown, an embodiment of the present invention further provides a computer device, including:

[0188] at least one processor 901;

[0189] At least one memory 902, used to store at least one program;

[0190] When at least one program is executed by at least one processor, the at least one processor implements Figure 1 The method shown.

[0191] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0192] The embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, Figure 1 The method shown.

[0193] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0194] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A lung image segmentation method, characterized in that: The following steps are involved: obtain lung images and radiology report information; Preprocessing the lung image to obtain initial image spatial features and initial image channel features; Preprocessing the radiology report information to obtain initial spatial level text features and initial channel level text features; According to the initial image spatial features, the initial image channel features, the initial spatial level text features and the initial channel level text features, a target cosine loss function is used to perform multimodal feature alignment processing to obtain a target alignment feature, wherein the target alignment feature includes a target image spatial feature, a target image channel feature, a target spatial level text feature and a target channel level text feature; Constructing a spatial-level multimodal feature according to the target image spatial feature and the target spatial-level text feature; Constructing a channel-level multimodal feature according to the target image channel feature and the target channel-level text feature; Constructing a target multimodal feature according to the dual cross-modal self-attention mechanism, the spatial-level multimodal feature, and the channel-level multimodal feature; According to the preset semantic level number, the target multimodal features at different semantic levels are adaptively gated and fused to obtain a lung image segmentation result.

2. The method according to claim 1, characterized in that The preprocessing of the lung image to obtain initial image spatial features and initial image channel features includes: extracting an initial feature map from the lung image; extracting visual features from the initial feature map; Performing L2 normalization on the visual features; According to the feature map size, pixel coordinates and preset spatial coordinate feature dimensions, construct a spatial coordinate feature corresponding to each pixel in the initial feature map, wherein the spatial coordinate feature includes horizontal position information of a first vector dimension, vertical position information of a second vector dimension, image width information of a third vector dimension and image height information of a fourth vector dimension; Performing L2 normalization on the spatial coordinate features; Constructing the initial image spatial features according to the spatial coordinate features normalized by the L2 paradigm and the visual features normalized by the L2 paradigm; The visual features normalized by the L2 paradigm are subjected to representation transformation processing to obtain the initial image channel features.

3. The method according to claim 2, characterized in that The preprocessing of the radiology report information to obtain initial spatial level text features and initial channel level text features includes: Performing word division processing on the radiology report information to obtain a word set; One-hot encoding is performed on the word set to obtain multiple one-hot vectors; Perform word embedding processing on the one-hot vector to obtain a word vector; According to the image space feature dimension of the initial image space feature, performing a first linear transformation on the word vector so that the feature dimension of the word vector after the first linear transformation is consistent with the image space feature dimension; Performing the L2 normalization on the word vector after the first linear transformation to obtain the initial spatial level text feature; According to the image channel feature dimension of the initial image channel feature, performing a second linear transformation on the word vector, so that the feature dimension of the word vector after the second linear transformation is consistent with the image channel feature dimension; The word vector after the second linear transformation is normalized using the L2 normal form to obtain the initial channel-level text features.

4. The method according to claim 1, characterized in that The method of performing multimodal feature alignment processing using a target cosine loss function according to the initial image space feature, the initial image channel feature, the initial space-level text feature, and the initial channel-level text feature to obtain a target alignment feature includes: Constructing an image spatial average feature according to the initial image spatial feature and the number of pixels in the feature map; Constructing an image channel average feature according to the initial image channel feature and the number of feature map channels; Constructing an average text feature at the spatial level according to the initial spatial level text feature and the number of text words; Constructing a channel-level text average feature according to the initial channel-level text feature and the number of text words; Constructing a spatial-level cosine loss function according to the spatial average feature of the image and the spatial-level text average feature; Constructing a channel-level cosine loss function according to the image channel average feature and the channel-level text average feature; Constructing the target cosine loss function according to the spatial level cosine loss function and the channel level cosine loss function; The target cosine loss function is used to update the initial image space features, the initial image channel features, the initial space-level text features and the initial channel-level text features to obtain the target alignment features.

5. The method according to claim 1, characterized in that The step of constructing a spatial-level multimodal feature according to the target image spatial feature and the target spatial-level text feature includes: Combining the target image spatial feature and the target spatial level text feature to obtain a spatial level cross-modal joint feature; According to the spatial-level cross-modal joint features, a query vector, a key vector, and a value vector are constructed respectively; constructing an attention score based on the query vector and the key vector; Constructing a value weighted vector according to the attention score and the value vector; According to the linear transformation weight, performing a third linear transformation on the value weighted vector so that the feature dimension of the value weighted vector is consistent with the feature dimension of the cross-modal joint feature at the spatial level; Constructing a spatial-level cross-modal self-attention feature according to the value weighted vector after the third linear transformation and the spatial-level cross-modal joint feature; The spatial-level multimodal features are constructed based on the spatial-level cross-modal self-attention features, and the channel dimension of the spatial-level multimodal features after the fourth linear transformation is 500 dimensions.

6. The method according to claim 1, characterized in that The constructing of channel-level multimodal features according to the target image channel features and the target channel-level text features includes: Combining the target image channel feature and the target channel-level text feature to obtain a channel-level cross-modal joint feature; Constructing an attention map according to the channel-level cross-modal joint features; Constructing a channel-level cross-modal self-attention feature according to the attention map and the channel-level cross-modal joint feature; The channel-level multimodal features are constructed based on the channel-level cross-modal self-attention features, and the channel dimension of the channel-level multimodal features after the fifth linear transformation is 500 dimensions.

7. The method according to claim 1, characterized in that The method of performing adaptive gating fusion processing on the target multimodal features at different semantic levels according to the preset semantic level number to obtain a lung image segmentation result includes: According to the semantic level, memory gate, storage gate and update gate are constructed respectively; Constructing a context controller according to the memory gate, the storage gate, the first forget gate coefficient and the target multimodal feature; Constructing a multimodal feature map according to the context controller, the second forget gate coefficient, the update gate and the target multimodal feature; Constructing a probability map according to the preset number of semantic levels, the multimodal feature maps at different semantic levels, the first preset convolutional layer and the preset activation function; Constructing a binary cross entropy loss function based on the probability map, the set of image pixels, and the segmentation mask of the segmentation gold standard; Constructing a total loss function according to the binary cross entropy loss function, the target cosine loss function and the empirical hyperparameters; Updating the multimodal feature map using the total loss function; Update the probability map according to the updated plurality of multimodal feature maps; The lung image segmentation result is generated according to the updated probability map.

8. A lung image segmentation device, characterized in that: include: The first module is used to obtain lung images and radiology report information; The second module is used to preprocess the lung image to obtain initial image spatial features and initial image channel features; The third module is used to pre-process the radiology report information to obtain initial spatial level text features and initial channel level text features; A fourth module is used to perform multimodal feature alignment processing using a target cosine loss function according to the initial image spatial features, the initial image channel features, the initial spatial level text features and the initial channel level text features to obtain target alignment features, wherein the target alignment features include target image spatial features, target image channel features, target spatial level text features and target channel level text features; A fifth module is used to construct a spatial-level multimodal feature according to the target image spatial feature and the target spatial-level text feature; A sixth module is used to construct a channel-level multimodal feature according to the target image channel feature and the target channel-level text feature; A seventh module is used to construct a target multimodal feature according to the dual cross-modal self-attention mechanism, the spatial multimodal feature and the channel multimodal feature; The eighth module is used to perform adaptive gated fusion processing on the target multimodal features at different semantic levels according to a preset number of semantic levels to obtain a lung image segmentation result.

9. A computer device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images