A method for automatic recognition and segmentation of paranasal sinus images based on multi-level hint learning

Abstract

The application discloses a kind of based on multilevel prompt learning paranasal sinus image automatic identification and segmentation method, it is related to the technical field of computer vision and biological information engineering.The method comprises the following steps: obtaining the multilevel visual features of paranasal sinus;Based on multilevel prompt learning, a multi-view prompt learning model is constructed, and the context prompt, mixed multi-modal prompt and label prompt of the original paranasal sinus image are obtained using the multi-view prompt learning model to obtain the multi-view prompt of paranasal sinus;A paranasal sinus segmentation network is constructed, and the multilevel visual features of paranasal sinus and the multi-view prompt of paranasal sinus are used to obtain the paranasal sinus image automatic identification and segmentation result.The application can solve the problem of lacking medical prior knowledge and sufficient semantics, avoid the continuous adjustment of semantic space during optimization, and reduce the interference of task-independent cross-modal concepts, thereby improving the accuracy of paranasal sinus image automatic identification and segmentation result.

Description

Technical Field

[0001] This invention relates to the technical fields of computer vision and bioinformatics engineering, specifically to a method for automatic identification and segmentation of sinus images based on multi-level cue learning. Background Technology

[0002] In modern medicine, medical imaging technology plays an indispensable role as a crucial tool for disease diagnosis, treatment planning, and scientific research and teaching. With the rapid development of medical imaging technology, the acquisition of medical imaging data has become increasingly rich and complex, providing doctors with more detailed information about patients' tissues, organs, and lesions. However, this massive amount of medical imaging data also brings new challenges, particularly in image analysis and processing. Image segmentation is a key task in medical image analysis, aiming to separate and extract different regions or structures in an image so that doctors and researchers can more accurately identify lesion areas, assess the severity of lesions, and develop treatment plans. However, medical images may be affected by various noises and artifacts during acquisition, transmission, and storage, leading to image quality degradation, which greatly hinders image segmentation tasks. Summary of the Invention

[0003] To address the aforementioned shortcomings in the existing technology, this invention provides an automatic sinus image recognition and segmentation method based on multi-level cue learning.

[0004] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0005] An automatic sinus image recognition and segmentation method based on multi-level cue learning includes the following steps:

[0006] S1. Construct a multimodal feature extraction model and use it to extract global and regional visual features from the original sinus images to obtain multi-level visual features of the sinuses.

[0007] S2. Construct a multi-view cue learning model based on multi-level cue learning, and use the multi-view cue learning model to obtain context cue, hybrid multimodal cue and label cue of the original sinus image to obtain multi-view cue of the sinus;

[0008] S3. Construct a sinus segmentation network. Based on the sinus segmentation network, and using the multi-level visual features of the sinuses in step S1 and the multi-view cues of the sinuses in step S2, obtain the automatic recognition and segmentation results of the sinus images.

[0009] Furthermore, in step S1, the multimodal feature extraction model includes a global-level visual feature branch, a regional-level visual feature branch, and a multi-level feature aggregation layer that is simultaneously connected to the outputs of both the global-level visual feature branch and the regional-level visual feature branch.

[0010] Furthermore, the global visual feature branch uses a pre-trained ViT as the encoder; the global visual feature branch segments the input raw sinus image into non-overlapping image patches and generates image patch embeddings and classification embeddings to extract the global visual features of the raw sinus image, represented as:

[0011]

[0012] in: These are global-level visual features of the original sinus images. For special Token classification embedding, Embedded for the first image patch, Embedded for the second image patch, For the first Image patch embedding, The number of image patches, For shape A two-dimensional array, The dimension of the image patch embedding.

[0013] Furthermore, the region-level visual feature branch uses the pre-trained Faster-RCNN toolkit; the region-level visual feature branch extracts the target object based on the confidence of the original sinus image, encodes the target object using object embedding, and transforms the encoded target object to match the dimension of the global-level visual features to extract the region-level visual features of the original sinus image, represented as:

[0014]

[0015]

[0016] in: The first image in the original sinus images, sorted in descending order of confidence level. Embedding of the target object corresponding to each object. , This is the first-dimensional transformation matrix. This is the second-dimensional transformation matrix. The first image in the original sinus images, sorted in descending order of confidence level. One object, This is the offset in the second dimension. This is the offset in the first dimension. For shape A one-dimensional array, For shape A two-dimensional array, The dimension of the image patch embedding. Regional visual features of the original sinus image. Embed the target object corresponding to the first object in the original sinus image, sorted in descending order of confidence. Embed the target object corresponding to the second object in the original sinus image, sorted in descending order of confidence. The first image in the original sinus images, sorted in descending order of confidence level. Embedding of the target object corresponding to each object. This is a hyperparameter that specifically represents the number of objects with high confidence.

[0017] Furthermore, the data processing procedure for the multi-level feature aggregation layer is represented as follows:

[0018]

[0019] in: The multi-layered visual features of the sinuses. To balance the coefficients of global-level visual features and regional-level visual features, These are global-level visual features of the original sinus images. Regional visual features of the original sinus image. For shape A two-dimensional array, The number of image patches, This is a hyperparameter, specifically representing the number of objects with high confidence. The dimension of the image patch embedding.

[0020] Furthermore, in step S2, the multi-view prompting learning model includes a description generation layer, a discrete coding layer, and an aggregation module connected in sequence.

[0021] Furthermore, the description generation layer is used to generate image descriptions based on the input raw sinus image.

[0022] Furthermore, the discrete coding layer is used to discretely encode the image description, obtaining discrete coded data of the image description; the data processing procedure of the discrete coding layer is represented as follows:

[0023]

[0024] in: Discrete encoded data for image description. For initialization function, For the RoBERTa network model, For image description, For shape A two-dimensional array, To control the hyperparameters of the number of mixed cues, The dimension that describes the image.

[0025] Furthermore, the aggregation module includes a low-aggregation layer, a mixed aggregation layer, and a high-aggregation layer connected in sequence; the data processing procedure of the mixed aggregation layer is represented as follows:

[0026]

[0027]

[0028] in: This is a polymerization method for hybrid polymer layers. For image description, Input the first image grid. For the second image grid input, The consistency score between the two modes. It is a vector The One element, It is a vector The One element, It is a vector The One element, It is a vector The One element, The final feature after the mapping dimension mapping function, For dimension mapping function, As the first text feature, For shape A two-dimensional array, To control the number of HEPs, The dimension that describes the image.

[0029] Further, in step S3, the sinus segmentation network includes a multi-level visual feature input layer, a multi-view cue input layer, a graph encoding layer, a text encoding layer, a cross-attention module, a U-Net network, and an output layer. The multi-level visual feature input layer is used to input multi-level visual features, and its output is connected to the input of the graph encoding layer, which in turn is connected to the input of the cross-attention module. The multi-view cue input layer is used to input multi-view cues, and its output is connected to the input of the text encoding layer, which in turn is connected to both the input of the cross-attention module and the input of the output layer. The output of the cross-attention module is connected to the input of the U-Net network, and the output of the U-Net network is connected to the input of the output layer.

[0030] The present invention has the following beneficial effects:

[0031] (1) This invention constructs a multimodal feature extraction model, uses the multimodal feature extraction model to extract global and regional visual features of the original sinus image to obtain multi-level visual features of the sinus, and then constructs a multi-view cue learning model based on multi-level cue learning. The multi-view cue learning model is used to obtain context cue, hybrid multimodal cue and label cue of the original sinus image to obtain multi-view cue of the sinus. Finally, a sinus segmentation network is constructed. Based on the sinus segmentation network, the multi-level visual features of the sinus and the multi-view cue of the sinus are used to obtain the automatic recognition and segmentation results of the sinus image, which can improve the accuracy of the automatic recognition and segmentation results of the sinus image.

[0032] (2) The present invention constructs a multi-view prompting learning model based on multi-level prompting learning. The multi-view prompting learning model is used to obtain contextual prompts, hybrid multimodal prompts and label prompts of the original sinus image to obtain multi-view prompts of the sinus. This can solve the problem of lack of medical prior knowledge and sufficient semantics, avoid the continuous adjustment of semantic space during the optimization process, and reduce the interference of task-irrelevant cross-modal concepts, thereby improving the accuracy of automatic recognition and segmentation results of sinus images. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of a method for automatic identification and segmentation of sinus images based on multi-level cue learning. Detailed Implementation

[0034] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0035] like Figure 1 As shown, an automatic sinus image recognition and segmentation method based on multi-level cue learning includes steps S1-S3, as detailed below:

[0036] S1. Construct a multimodal feature extraction model and use it to extract global and regional visual features from the original sinus images to obtain multi-level visual features of the sinuses.

[0037] In an optional embodiment of the present invention, the multimodal feature extraction model includes a global-level visual feature branch, a regional-level visual feature branch, and a multi-level feature aggregation layer that is simultaneously connected to the outputs of the global-level visual feature branch and the regional-level visual feature branch.

[0038] The global visual feature branch uses a pre-trained ViT as the encoder; the global visual feature branch segments the input raw sinus image into non-overlapping image patches and generates image patch embeddings and classification embeddings to extract the global visual features of the raw sinus image, represented as:

[0039]

[0040] in: These are global-level visual features of the original sinus images. For special Token classification embedding, Embedded for the first image patch, Embedded for the second image patch, For the first Image patch embedding, The number of image patches, For shape A two-dimensional array, The dimension of the image patch embedding.

[0041] The regional visual feature branch uses the pre-trained Faster-RCNN toolkit. Based on the confidence level of the original sinus image, the branch extracts the target object, encodes it using object embedding, and transforms the encoded target object to match the dimensions of the global visual features to extract the regional visual features of the original sinus image, represented as follows:

[0042]

[0043]

[0044] in: The first image in the original sinus images, sorted in descending order of confidence level. Embedding of the target object corresponding to each object. , This is the first-dimensional transformation matrix. This is the second-dimensional transformation matrix. The first image in the original sinus images, sorted in descending order of confidence level. One object, This is the offset in the second dimension. This is the offset in the first dimension. For shape A one-dimensional array, For shape A two-dimensional array, The dimension of the image patch embedding. Regional visual features of the original sinus image. Embed the target object corresponding to the first object in the original sinus image, sorted in descending order of confidence. Embed the target object corresponding to the second object in the original sinus image, sorted in descending order of confidence. The first image in the original sinus images, sorted in descending order of confidence level. Embedding of the target object corresponding to each object. This is a hyperparameter that specifically represents the number of objects with high confidence.

[0045] The data processing procedure for the multi-level feature aggregation layer is represented as follows:

[0046]

[0047] in: The multi-layered visual features of the sinuses. To balance the coefficients of global-level visual features and regional-level visual features, These are global-level visual features of the original sinus images. Regional visual features of the original sinus image. For shape A two-dimensional array, The number of image patches, This is a hyperparameter, specifically representing the number of objects with high confidence. The dimension of the image patch embedding.

[0048] S2. Construct a multi-view cue learning model based on multi-level cue learning, and use the multi-view cue learning model to obtain contextual cue, hybrid multimodal cue and label cue of the original sinus image to obtain multi-view cue of the sinus.

[0049] In an optional embodiment of the present invention, the multi-view prompting learning model includes a description generation layer, a discrete encoding layer, and an aggregation module connected in sequence.

[0050] The description generation layer is used to generate image descriptions based on the input raw sinus image. This invention uses the existing ClipCap (an image caption generation toolkit) to obtain image descriptions of the input raw sinus image.

[0051] The discrete coding layer is used to perform discrete coding on the image description, obtaining discrete coded data of the image description; the data processing process of the discrete coding layer is represented as follows:

[0052]

[0053] in: Discrete encoded data for image description. For initialization function, For the RoBERTa network model, For image description, For shape A two-dimensional array, To control the hyperparameters of the number of mixed cues, The dimension that describes the image.

[0054] The aggregation module consists of a low-aggregation layer, a mixed aggregation layer, and a high-aggregation layer connected in sequence; the data processing procedure of the mixed aggregation layer is represented as follows:

[0055]

[0056]

[0057] in: This is a polymerization method for hybrid polymer layers. For image description, Input the first image grid. For the second image grid input, The consistency score between the two modes. It is a vector The One element, It is a vector The One element, It is a vector The One element, It is a vector The One element, The final feature after the mapping dimension mapping function, For dimension mapping function, As the first text feature, For shape A two-dimensional array, To control the number of HEPs, The dimension that describes the image.

[0058] This invention calculates the consistency score between two modes, expressed as:

[0059]

[0060] in: For text data and visual data Consistency score, for Activation function for Activation function.

[0061] S3. Construct a sinus segmentation network. Based on the sinus segmentation network, and using the multi-level visual features of the sinuses in step S1 and the multi-view cues of the sinuses in step S2, obtain the automatic recognition and segmentation results of the sinus images.

[0062] In an optional embodiment of the present invention, the sinus segmentation network includes a multi-level visual feature input layer, a multi-view cue input layer, a graph encoding layer, a text encoding layer, a cross-attention module, a U-Net network, and an output layer. The multi-level visual feature input layer is used to input multi-level visual features, and its output is connected to the input of the graph encoding layer, which in turn is connected to the input of the cross-attention module. The multi-view cue input layer is used to input multi-view cues, and its output is connected to the input of the text encoding layer, which in turn is connected to both the input of the cross-attention module and the input of the output layer. The output of the cross-attention module is connected to the input of the U-Net network, which in turn is connected to the input of the output layer.

[0063] The image coding layer encodes multi-level visual features, obtaining the encoded data of these multi-level visual features, represented as follows:

[0064]

[0065] in: Encoded data for multi-level visual features, Special markings for classification, The first element of the hierarchical visual features. The second element of the hierarchical visual features. The first of the hierarchical visual features One element, This represents the number of hierarchical visual features.

[0066] The text encoding layer encodes the multi-view cues, obtaining the encoded data of the multi-view cues, represented as:

[0067]

[0068] in: Encoded data for multi-view prompts, The first element for multi-view prompts, The second element for multi-view prompts, The first point of view for multiple perspectives One element, The number of multi-view prompts.

[0069] The data processing procedure of the cross-attention module is represented as follows:

[0070]

[0071]

[0072]

[0073] in: As a feature of fusion, For layer normalization operation, for, As a second text feature, The mean of the fusion features, For average pooling operation, The number of multi-view prompts, For conditional probability, This is the cosine similarity calculation function. As a third text feature, This is the temperature coefficient.

[0074] The data processing procedure of the U-Net network is as follows:

[0075]

[0076]

[0077] in: To output feature map at position The value, For the sample size, For position Image features, For position The weight matrix, This is the offset. For the input feature map in the region (usually) The value within the window. The sign for the maximum value.

[0078] This invention optimizes the sinus segmentation network by calculating the cross-entropy loss function, expressed as:

[0079]

[0080] in: This is the cross-entropy loss function.

[0081] This invention obtains a sinus tumor dataset, and uses this dataset to obtain the automatic identification and segmentation results of the method of this invention and other methods, as shown in the table below:

[0082]

[0083] Here, mDice is a set similarity metric used to calculate the similarity between two samples; mIoU is the mean intersection-over-union ratio, a detection metric used to measure the average intersection-over-union ratio of a network; and MAE is the mean absolute error, a metric used to measure the accuracy of a prediction model or estimation method in statistics. The simulation results show that this invention, by constructing a multimodal feature extraction model, extracts global and regional visual features from the original sinus images to obtain multi-level visual features of the sinuses. Then, based on multi-level cue learning, a multi-view cue learning model is constructed to obtain contextual cues, hybrid multimodal cues, and label cues from the original sinus images to obtain multi-view cueing for the sinuses. Finally, a sinus segmentation network is constructed. Based on this network, and utilizing the multi-level visual features and multi-view cueing of the sinuses, the automatic identification and segmentation results of sinus images are obtained, which can improve the accuracy of automatic sinus image identification and segmentation.

[0084] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0087] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0088] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for automatic identification and segmentation of sinus images based on multi-level cue learning, characterized in that, Includes the following steps: S1. Construct a multimodal feature extraction model to extract global and regional visual features from the original sinus image, thereby obtaining multi-level visual features of the sinuses. The multimodal feature extraction model includes a global visual feature branch, a regional visual feature branch, and a multi-level feature aggregation layer connected to the outputs of both the global and regional visual feature branches. The global visual feature branch uses a pre-trained ViT as the encoder; the global visual feature branch segments the input raw sinus image into non-overlapping image patches and generates image patch embeddings and classification embeddings to extract the global visual features of the raw sinus image, represented as: in: These are global-level visual features of the original sinus images. For special Token classification embedding, Embedded for the first image patch, Embedded for the second image patch, For the first Image patch embedding, The number of image patches, For shape A two-dimensional array, The dimension of the image patch embedding; The regional visual feature branch uses the pre-trained Faster-RCNN toolkit. Based on the confidence level of the original sinus image, the branch extracts the target object, encodes it using object embedding, and transforms the encoded target object to match the dimensions of the global visual features to extract the regional visual features of the original sinus image, represented as follows: in: The first image in the original sinus images, sorted in descending order of confidence level. Embedding of the target object corresponding to each object. , This is the first-dimensional transformation matrix. This is the second-dimensional transformation matrix. The first image in the original sinus images, sorted in descending order of confidence level. One object, This is the offset in the second dimension. This is the offset in the first dimension. For shape A one-dimensional array, For shape A two-dimensional array, The dimension of the image patch embedding. Regional visual features of the original sinus image. Embed the target object corresponding to the first object in the original sinus image, sorted in descending order of confidence. Embed the target object corresponding to the second object in the original sinus image, sorted in descending order of confidence. The first image in the original sinus images, sorted in descending order of confidence level. Embedding of the target object corresponding to each object. This is a hyperparameter, specifically representing the number of objects with high confidence. The data processing procedure for the multi-level feature aggregation layer is represented as follows: in: The multi-layered visual features of the sinuses. To balance the coefficients of global-level visual features and regional-level visual features, These are global-level visual features of the original sinus images. Regional visual features of the original sinus image. For shape A two-dimensional array, The number of image patches, This is a hyperparameter, specifically representing the number of objects with high confidence. The dimension of the image patch embedding; S2. Construct a multi-view cue learning model based on multi-level cue learning, and use the multi-view cue learning model to obtain context cue, hybrid multimodal cue and label cue of the original sinus image to obtain multi-view cue of the sinus; S3. Construct a sinus segmentation network. Based on the sinus segmentation network, and using the multi-level visual features of the sinuses in step S1 and the multi-view cues of the sinuses in step S2, obtain the automatic recognition and segmentation results of the sinus images.

2. The method for automatic identification and segmentation of sinus images based on multi-level cue learning according to claim 1, characterized in that, In step S2, the multi-view prompting learning model includes a description generation layer, a discrete coding layer, and an aggregation module connected in sequence.

3. The method for automatic identification and segmentation of sinus images based on multi-level cue learning according to claim 2, characterized in that, The description generation layer is used to generate image descriptions based on the input raw sinus image.

4. The method for automatic identification and segmentation of sinus images based on multi-level cue learning according to claim 2, characterized in that, The discrete coding layer is used to perform discrete coding on the image description, obtaining discrete coded data of the image description; the data processing process of the discrete coding layer is represented as follows: in: Discrete encoded data for image description. For initialization function, For the RoBERTa network model, For image description, For shape A two-dimensional array, To control the hyperparameters for the number of mixed cues, The dimension that describes the image.

5. The method for automatic identification and segmentation of sinus images based on multi-level cue learning according to claim 2, characterized in that, The aggregation module consists of a low-aggregation layer, a mixed aggregation layer, and a high-aggregation layer connected in sequence; the data processing procedure of the mixed aggregation layer is represented as follows: in: This is a polymerization method for hybrid polymer layers. For image description, Input the first image grid. For the second image grid input, The consistency score between the two modes. It is a vector The One element, It is a vector The One element, It is a vector The One element, It is a vector The One element, The final feature after the mapping dimension mapping function, For dimension mapping function, As the first text feature, For shape A two-dimensional array, To control the number of HEPs, The dimension that describes the image.

6. The method for automatic identification and segmentation of sinus images based on multi-level cue learning according to claim 1, characterized in that, In step S3, the sinus segmentation network includes a multi-level visual feature input layer, a multi-view cue input layer, a graph encoding layer, a text encoding layer, a cross-attention module, a U-Net network, and an output layer. The multi-level visual feature input layer is used to input multi-level visual features, and its output is connected to the input of the graph encoding layer, which in turn is connected to the input of the cross-attention module. The multi-view cue input layer is used to input multi-view cues, and its output is connected to the input of the text encoding layer, which in turn is connected to both the input of the cross-attention module and the input of the output layer. The output of the cross-attention module is connected to the input of the U-Net network, and the output of the U-Net network is connected to the input of the output layer.

Images