Pulmonary tuberculosis screening method and system based on medical image information large model

Abstract

The invention provides a pulmonary tuberculosis screening method and device based on a medical image information large model, relates to the technical field of medical image information processing, and solves the technical problems that multi-source information cannot be fused and an interpretable diagnosis basis cannot be provided in the prior art. The method comprises the steps that a chest medical image is obtained and preprocessed, and a standardized image is obtained; obtaining a chest report and examination data, performing data preprocessing, and performing medical entity recognition to obtain structured text data; feature sequences of the standardized image and the structured text data are extracted respectively, and a fusion feature sequence is generated through fusion of a cross-modal attention algorithm; the feature sequence of the standardized image is extracted through a pre-trained ViT model; and inputting the fused feature sequence into a plurality of parallel task heads for parallel processing to obtain an individual screening result. The method is used in the pulmonary tuberculosis screening process.

Description

Technical Field

[0001] This application relates to the field of medical image information processing technology, and in particular to a method and system for tuberculosis screening based on a large model of medical image information. Background Technology

[0002] Pulmonary tuberculosis (TB) is a chronic infectious disease caused by Mycobacterium tuberculosis and is a major global public health problem. Early screening and diagnosis are crucial to stopping its spread and reducing mortality. Traditional imaging diagnosis, which relies heavily on radiologists' manual interpretation of images, results are heavily dependent on the physician's personal experience and knowledge. Early computer-aided diagnostic systems often used manual feature extraction combined with classifiers, but this method struggles to comprehensively and deeply capture the complex and subtle imaging features associated with TB, and is sensitive to changes in imaging equipment and scanning parameters, exhibiting significant performance degradation on datasets from different sources.

[0003] In recent years, the use of deep learning models such as CNNs for tuberculosis detection has become a research hotspot. Although their performance is superior to traditional methods, these models only provide classification results, failing to offer convincing medical evidence, and only complete the detection task without being able to locate lesions in the images or assess their severity. Therefore, this invention provides a tuberculosis screening method and system based on a large model of medical image information. Summary of the Invention

[0004] This application provides a method and device for tuberculosis screening based on a large model of medical imaging information, which solves the technical problem that existing technologies cannot integrate multi-source information and cannot provide interpretable diagnostic evidence.

[0005] To achieve the above objectives, this application adopts the following technical solution: Firstly, a method for tuberculosis screening based on a large model of medical image information is provided, including: acquiring and preprocessing chest medical images to obtain standardized images; acquiring chest reports and test data, performing data preprocessing, and obtaining structured text data through medical entity recognition; extracting feature sequences from the standardized images and structured text data respectively, and fusing them through a cross-modal attention algorithm to generate a fused feature sequence; the feature sequence of the standardized images is extracted using a pre-trained ViT model; the feature sequence of the structured text data is extracted using a pre-trained text Transformer model; the fused feature sequence is input into multiple parallel task heads for parallel processing to obtain individual screening results; wherein, the individual screening results include: tuberculosis classification, lesion localization map, and diagnostic report.

[0006] Based on the above technical solutions, the tuberculosis screening method based on a large medical image information model provided in this application, after structured processing of chest medical images, reports, and test data, uses a cross-modal attention algorithm to associate and complement standardized image features with structured text information, solving the limitations of single information data when recognizing images and improving the comprehensiveness of screening information. A pre-trained ViT model accurately extracts details of lesions in the images, and a pre-trained Transformer model captures medical descriptions related to tuberculosis in the reports. The fused feature sequence better aligns with the screening logic for tuberculosis, enhancing the accuracy of the screening results. A parallel task head synchronously outputs tuberculosis classification, lesion localization maps, and diagnostic reports, providing intuitive and traceable diagnostic evidence, solving the problem of lack of diagnostic support in existing technologies.

[0007] In conjunction with the first aspect mentioned above, one possible implementation method for obtaining structured text data includes: extracting chest report and test data, converting the chest report and test data into text data; removing text noise from the text data to obtain terminology text; and splitting and cleaning the terminology text using a medical-specific word segmenter to obtain medical vocabulary unit text; wherein, the medical vocabulary unit text is: splitting the terminology text into semantically complete medical vocabulary units.

[0008] The medical vocabulary unit text is input into the medical corpus model to obtain medical key entities. Through sequence labeling, the medical key entities are identified and their corresponding entity types are labeled. The medical corpus model is obtained by training a Transformer model with the boundaries of medical key entities in the medical vocabulary unit text and the medical key entities. The boundaries of medical key entities are the start and end positions of words or phrases with medical significance. The entity types are organized into structured text data according to a preset format.

[0009] By removing redundant information from reports and test data, the efficiency of medical vocabulary unit text acquisition is improved. A medical-specific word segmenter is used to avoid the misinterpretation of technical terms by ordinary word segmenters. Combined with a medical corpus model, key medical entities are identified, preserving the semantic integrity of medical data. Structured data that can be directly parsed by computers is generated, solving the problem that unstructured text cannot be directly used by computer models and providing standardized input for subsequent text feature extraction.

[0010] In conjunction with the first aspect mentioned above, one possible implementation involves obtaining the image feature sequence by dividing the standardized image I into N patches of size P×P; where P is the length of each patch. H is the height of the image, and W is the width of the image.

[0011] Convert the pixel matrix of each patch into a one-dimensional vector. The projection matrix E maps the one-dimensional vectors of all tiles to high-dimensional feature vectors. And concatenate the projection vectors of all tiles to form a classification token. and the addition of standard position coding The sequence format is obtained. ; where the high-dimensional feature vector is ( The sequence format is , It is a learnable classification token. It is the vector formed by the projection of the i-th tile, i∈[1,N].

[0012] sequence format Input ViT encoder, output global visual feature vector and local visual feature sequences , global visual feature vector and local visual feature sequences The data is integrated into an image feature sequence; wherein, the ViT encoder consists of an L-layer multi-head self-attention MSA, a multilayer perceptron MLP, and a layer normalization layer (LayerNorm), and the global visual feature vector output by the L-th layer is... The local visual feature sequence corresponding to the 1st to Nth patches output by the Lth layer is: .

[0013] By using location encoding and local visual feature sequences, the problem of insufficient spatial correlation in large-size medical images by traditional CNNs is solved, and the spatial location relationship of lesions is accurately restored. The overall information of the image is standardized by aggregating global visual feature vectors, and the details of the image are preserved by local visual feature sequences. The integration of the two makes the image features more comprehensive.

[0014] In conjunction with the first aspect mentioned above, one possible implementation is multi-head self-attention. The formula for calculation is: ; Where MSA() is the multi-head self-attention function, and LayerNorm() is the layer normalization function. This is the feature vector for each position in the matrix that incorporates information related to other positions. For the first The feature matrix of the layer after processing by the MSA sublayer and the MLP sublayer.

[0015] In conjunction with the first aspect mentioned above, in one possible implementation, the multilayer perceptron... The formula for calculation is: ; Where LayerNorm() is the layer normalization function, and MLP() is the multilayer perceptron function. This is the feature vector for each position in the matrix that incorporates information related to other positions. For the first Feature representation after layer processing.

[0016] In conjunction with the first aspect mentioned above, one possible implementation method for obtaining text feature sequences includes: using a medical-specific word segmenter to convert structured text data into a token sequence, inputting the token sequence into a Transformer model pre-trained on a medical corpus, and outputting a set of semantic feature vectors corresponding to each token in the last layer.

[0017] The output corresponding to the [CLS] (Classification Token) in the semantic feature vector set is used as the global text feature vector. The output of all word tokens in the semantic feature vector set is used as the local text feature sequence. , global text feature vector and local text feature sequences The features are integrated into a text feature sequence; where the [CLS] token is a fixed starting token preset by the model.

[0018] In conjunction with the first aspect mentioned above, one possible implementation involves obtaining the fusion feature sequence by: employing an attention mechanism to calculate the correlation between each patch and all medical vocabulary units, thereby obtaining an initial output feature sequence from the attention mechanism; and performing a dimensionality transformation on the initial output feature sequence, resulting in an output dimension that is equal to the given dimension. Consistent fusion of local fusion feature sequences Based on [CLS] tokens or pairs Pooling is performed to obtain the global fused feature vector. Globally fuse feature vectors With local fusion feature sequences The sequences are integrated into a fusion feature sequence.

[0019] By employing a cross-modal attention mechanism, the relevance between image patches and medical vocabulary units can be directly calculated, thereby improving the semantic consistency of fused features.

[0020] In conjunction with the first aspect mentioned above, one possible implementation involves inputting the fused feature sequence into multiple parallel task heads for parallel processing to obtain individual screening results, including: processing the global visual feature vector... With global text feature vector The combined features are obtained by splicing. The input is then fed into a fully connected layer for feature mapping, outputting a two-dimensional vector. The probability distribution of this two-dimensional vector is calculated using the Softmax function to obtain the positive / negative probability of tuberculosis. The formula for calculating the probability distribution is Pcls = softmax(FC( )).

[0021] Local fusion feature sequences Reconstructing the original image to its corresponding two-dimensional spatial layout The decoder upsamples the F2D layer by layer to the original image resolution, and then uses the Sigmoid activation function to generate a lesion segmentation mask. .

[0022] The gradient of the positive category in the pulmonary tuberculosis positive / negative classification task is calculated based on the Grad-CAM algorithm, and the fused local fusion feature sequence is then analyzed. Gradient-weighted summation is performed to obtain a coarse heatmap. This coarse heatmap is then upsampled to the original image size using bilinear interpolation to obtain a class activation heatmap. .

[0023] Based on the lesion segmentation mask The corresponding lesion location, size, and morphology information are then fused with the global feature vector. After integration, the data is input into an autoregressive decoder to generate a structured diagnostic report (RR) that matches the lesion information.

[0024] Based on the above technical solution, in the tuberculosis screening method based on a large medical image information model provided in this application, the pixel range of the lesion is marked by the lesion segmentation mask, which solves the problem that the traditional model only provides the result but not the location. Thus, the lesion can be quickly located in the tuberculosis screening process.

[0025] In conjunction with the first aspect above, in one possible implementation, the method for acquiring standardized images includes: extracting the chest medical image; wherein the chest medical image includes: X-ray image or CT image; performing standardization processing on the chest medical image to obtain a standardized image; wherein the standardization processing includes: grayscale normalization, resampling to a uniform size, and data augmentation.

[0026] Secondly, this application provides a tuberculosis screening system based on a large medical imaging information model, comprising: a standardized image module, a structured text data module, a fusion feature sequence module, and an individual screening result module; wherein, the standardized image module is used to acquire chest medical images and perform preprocessing to obtain standardized images; the structured text data module is used to acquire chest reports and test data, perform data preprocessing, and then obtain structured text data through medical entity recognition; the fusion feature sequence module is used to extract feature sequences from the standardized images and structured text data respectively, and fuse them to generate a fused feature sequence through a cross-modal attention algorithm; the feature sequence of the standardized images is extracted through a pre-trained ViT model; the feature sequence of the structured text data is extracted through a pre-trained text Transformer model; the individual screening result module is used to input the fused feature sequence into multiple parallel task heads for parallel processing to obtain individual screening results; wherein, the individual screening results include: tuberculosis classification, lesion localization map, and diagnostic report.

[0027] This application provides a method and device for tuberculosis screening based on a large model of medical imaging information. It can fuse the feature sequences of standardized image data and structured text data using a cross-modal attention algorithm to generate a fused feature sequence. This allows the model to leverage the complementarity of image and text information, reducing the ambiguity of a single modality, thereby achieving higher diagnostic accuracy in complex and atypical cases and demonstrating better generalization ability to data from different sources. The provided lesion localization map and automatically generated text report solve the technical problem that existing technologies cannot fuse multi-source information and cannot provide interpretable diagnostic evidence.

[0028] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0029] Figure 1 A system structure diagram of a tuberculosis screening system based on a large model of medical imaging information provided in this application embodiment; Figure 2 A flowchart illustrating a tuberculosis screening method based on a large medical imaging information model, provided for an embodiment of this application; Figure 3 A flowchart illustrating another tuberculosis screening method based on a large medical imaging information model provided in this application embodiment; Figure 4 A flowchart illustrating another tuberculosis screening method based on a large medical imaging information model provided in this application embodiment; Figure 5 A flowchart illustrating another tuberculosis screening method based on a large medical imaging information model provided in this application embodiment; Detailed Implementation

[0030] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0031] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0032] The tuberculosis screening method based on a large medical imaging information model provided in this application can be applied to, for example... Figure 1 The tuberculosis screening system based on a large model of medical imaging information shown below, such as Figure 1 As shown, the system includes: a standardized image module, a structured text data module, a fused feature sequence module, and an individual screening result module.

[0033] The system comprises several modules: a standardized image module for acquiring and preprocessing chest medical images to obtain standardized images; a structured text data module for acquiring chest reports and test data, preprocessing the data, and then generating structured text data through medical entity recognition; a fusion feature sequence module for extracting feature sequences from standardized images and structured text data respectively, and then fusing them using a cross-modal attention algorithm to generate a fused feature sequence; the feature sequence of standardized images is extracted using a pre-trained ViT model, and the feature sequence of structured text data is extracted using a pre-trained text Transformer model; and an individual screening result module for inputting the fused feature sequence into multiple parallel task heads for parallel processing to obtain individual screening results, including: tuberculosis classification, lesion localization map, and diagnostic report.

[0034] To address the technical problem that existing technologies cannot integrate multi-source information and provide interpretable diagnostic evidence, this application provides a tuberculosis screening method based on a large model of medical image information. The method includes: acquiring and preprocessing chest medical images to obtain standardized images; acquiring chest reports and test data, performing data preprocessing, and then obtaining structured text data through medical entity recognition; extracting feature sequences from the standardized images and structured text data respectively, and fusing them using a cross-modal attention algorithm to generate a fused feature sequence; the feature sequence of the standardized images is extracted using a pre-trained ViT model; the feature sequence of the structured text data is extracted using a pre-trained text Transformer model; and the fused feature sequence is input into multiple parallel task heads for parallel processing to obtain individual screening results; wherein, the individual screening results include: tuberculosis classification, lesion localization map, and diagnostic report.

[0035] like Figure 2 As shown in the embodiments of this application, the tuberculosis screening method based on a large model of medical imaging information includes: S201. Acquire chest medical images and perform preprocessing to obtain standardized images.

[0036] Chest medical images include X-ray or CT images; standardization processing includes grayscale normalization, resampling to a uniform size, and data augmentation.

[0037] It should be noted that due to differences in imaging equipment, the grayscale value range of chest medical images varies greatly. Therefore, it is necessary to normalize the grayscale values ​​to make the image data distribution more stable and avoid misjudging features by the model due to grayscale differences. Data augmentation of chest medical images can increase the robustness of the model.

[0038] S202. After obtaining chest reports and test data and performing data preprocessing, structured text data is obtained through medical entity recognition.

[0039] Structured text data refers to structured data that can be directly parsed by computers.

[0040] It should be noted that text data may contain issues such as disordered formatting and redundant characters. Therefore, it is necessary to clean the text data to unify the text format and obtain medical terminology unit text. In order to ensure the semantic integrity of medical terms, support the recognition of key medical entities, and improve the matching degree of cross-modal fusion, it is necessary to ensure the semantic integrity of medical terms, support the recognition of key medical entities, and improve the matching degree of cross-modal fusion.

[0041] For example, when using medical-specific word segmentation to split terminology text, such as splitting "cavitary shadow in the upper lobe of the left lung" into "cavitary shadow in the upper lobe of the left lung", it can avoid the common word segmenter splitting professional terms such as "cavitary shadow".

[0042] S203. Extract the feature sequences of standardized image and structured text data respectively, and generate a fused feature sequence by fusing them through a cross-modal attention algorithm.

[0043] The feature sequences of the standardized images were extracted using a pre-trained ViT model; the feature sequences of the structured text data were extracted using a pre-trained text Transformer model. , , .

[0044] In some implementations, a medical-specific word segmenter is used to convert structured text data into a token sequence. This token sequence is then input into a Transformer model pre-trained on a medical corpus, outputting a set of semantic feature vectors corresponding to each token in the last layer. The output corresponding to the [CLS] token in this set of semantic feature vectors is used as the global text feature vector. The output of all word tokens in the semantic feature vector set is used as the local text feature sequence. , global text feature vector and local text feature sequences The features are integrated into a text feature sequence; where the [CLS] token is a fixed starting token preset by the model.

[0045] An attention mechanism is used to calculate the correlation between each image patch and all medical vocabulary units, resulting in an initial output feature sequence. The initial output feature sequence is then dimensionality-transformed, with the output dimension being equal to... Consistent fusion of local fusion feature sequences Based on [CLS] tokens or pairs Pooling is performed to obtain the global fused feature vector. Globally fuse feature vectors With local fusion feature sequences The sequences are integrated into a fusion feature sequence.

[0046] In some implementations, the attention weights are calculated as follows:

[0047] Where Q is the query vector, K is the key vector, and V is the value vector. This indicates the degree of association between each tile and each medical terminology unit. These are the feature dimensions of Q and K; ,in, , , It is a learnable projection matrix.

[0048] It should be noted that the global fusion feature vector The purpose of inputting the autoregressive decoder is to generate a structured diagnostic report with global semantic integrity and stable features; therefore, this method uses average pooling.

[0049] It should be noted that each element in the initial output feature sequence of the attention mechanism is no longer a simple visual feature of an image patch, but a cross-modal local feature that simultaneously contains visual details of the patch and matching medical vocabulary units. Each vector in this sequence incorporates the most relevant textual information, but it is not the final fused feature sequence.

[0050] It should also be noted that the projection matrix is ​​a real number matrix with a fixed shape, which can transform a feature vector of one dimension into a feature term vector of another dimension. The learnable projection matrix, on the other hand, is a parameter matrix in a deep learning model. All the values ​​of the learnable projection matrix are not manually set, nor are they fixed. Instead, each value in the matrix is ​​continuously adjusted according to the algorithm.

[0051] For example, in the embodiments of this application, local visual feature sequences yes The matrix, the learnable projection matrix yes The matrix, where, It is the input dimension, matching The number of columns, This is the output dimension, matching the dimension of text K. It involves the local visual feature vector. With learnable projection matrix Multiplying them together gives the dimension as query vector , The multiplication process of dimensions changes as follows .

[0052] For example, in this embodiment of the application, taking chest X-ray images and electronic chest reports as examples, assume the data to be screened is: Chest X-ray image: a grayscale image with a resolution of 1024×1024 pixels and a channel number C=1; Electronic medical record text: Cough for 2 weeks, patchy shadows seen in the upper lobe of the left lung; The first step is to obtain local visual feature sequences. The process is divided into image segmentation and ViT feature extraction: the 1024×1024 X-ray image is segmented into P=16×16 patches, with the number of patches N=(1024×1024) / (16×16)=4096; ViT feature extraction: local visual features are extracted using a pre-trained ViT model. Its dimensions are N×d=4096×768, where d=768 is the feature dimension of a single tile.

[0053] The second step is to obtain local text feature sequences. It is divided into text tokenization and Transformer feature extraction: The text is converted into a token sequence using a medical tokenizer: [CLS], cough, 2 weeks, left upper lobe, see, patchy shadow, [SEP], so M is 7 tokens; local text feature sequences are extracted using a pre-trained ClinicalBERT model. Its dimensions are .

[0054] The third step is to generate the Q, K, and V parameters for cross-modal attention: Set the output dimensions of the projection matrix =512, used to match the interaction dimension between text and visual features; generates a query vector Q, a learnable projection matrix. The dimension is =768 512, then Q= =(4096 ) (768 512)=4096 512; Generate key vector K: a learnable projection matrix The dimension is 768 512, then K= =( ) (768 512)=7 512; Generated value vector K: Learnable projection matrix The dimension is 768 512, V= =( ) (768 512)=7 512.

[0055] The third step is to calculate the cross-modal attention weights and output: Attention score is calculated as follows: =(4096 512) (512 7) = 4096 7; Scaling and Softm ax normalized to: = =4096 7; Attention output is = =(4096 7)(7 512)=4096 512.

[0056] S204. Input the fused feature sequence into multiple parallel task heads for parallel processing to obtain individual screening results.

[0057] Individual screening results include: tuberculosis classification, lesion localization map, and diagnostic report.

[0058] It should be noted that the sub-task head types of the task head are classification task head, segmentation / localization task head, category activation heatmap task head, and report generation task head; existing technologies mostly use single-modal features, while this method is a cross-modal fusion feature of standardized image and structured text data, which has more comprehensive task coverage and fully matches the needs of the entire process of tuberculosis screening. The report generation task head is also more in line with the medical standard description of tuberculosis diagnosis.

[0059] Based on the above technical solution, the tuberculosis screening method based on a large medical imaging information model provided in this application deeply integrates visual and textual information through multimodal feature extraction and fusion based on the large model, forming a joint representation that can mutually enhance each other. Screening, localization, and report generation are completed simultaneously through a single model, simplifying the clinical workflow and improving efficiency. This solves the technical problem that existing technologies cannot integrate multi-source information and cannot provide interpretable diagnostic evidence.

[0060] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, step S202 can be implemented through steps S301-S305, which are explained in detail below: S301. Extract the chest report and test data, and convert the chest report and test data into text data.

[0061] In some implementations, if the chest report is electronic, its content can be directly read through the hospital information system's interface to obtain digitized text; if it is a paper report, optical character recognition technology can be used to scan the paper document, converting printed or handwritten text into editable digitized text data. The test data, including test items, results, and reference values, is exported from the hospital information system. A text conversion script then transforms the structured test data into a natural language text description, which is then merged with the chest report's text data.

[0062] It should be noted that chest reports may be in electronic or paper form, and test data are usually in tabular or code format. If it is not a standardized text format, it cannot be directly input into a medical-specific word segmenter and medical corpus model.

[0063] S302. Remove text noise from the text data to obtain the terminology text.

[0064] Among them, text noise refers to redundant information in the text data that is unrelated to medical content.

[0065] In some implementations, irrelevant content is matched and deleted by pre-defined rules. Then, based on a medical terminology dictionary, the content in the text that matches medical terms is retained, while non-medical words outside the dictionary are filtered out. Finally, a lightweight text classification model is used to distinguish between medical-related content and noisy content, and text fragments containing medical terms are automatically selected.

[0066] It should be noted that the terminology text obtained after noise removal retains only the medical-related content, which reduces the processing burden of the medical-specific word segmenter, avoids irrelevant information from interfering with the integrity of medical vocabulary unit segmentation, and lays an accurate text foundation for medical key entity recognition and text feature extraction.

[0067] For example, removing non-medical redundant text such as hospital names, doctor signatures, and "hello" from the report, as well as extra spaces, punctuation marks, and garbled characters.

[0068] S303. The terminology text is split and cleaned using a medical-specific word segmenter to obtain medical vocabulary unit text.

[0069] Among them, the medical vocabulary unit text is: the terminology text is broken down into semantically complete medical vocabulary units.

[0070] In some implementations, a professional medical terminology dictionary is first constructed, which includes anatomical locations, lesion types, and laboratory test terms related to tuberculosis. Then, a word segmentation model is trained or adapted based on this dictionary. The denoised terminology text is input into a word segmenter, which, based on the medical dictionary, splits the text into semantically complete medical vocabulary units. Invalid units generated during the segmentation process, such as repeated punctuation marks and redundant words without medical meaning, are filtered out, and finally, the text containing only valid medical vocabulary units is obtained.

[0071] It should be noted that ordinary word segmenters are trained on general corpora and may incorrectly segment medical terms, resulting in the destruction of medical semantics. In contrast, medical-specific word segmenters can recognize and preserve the integrity of medical terms, preventing entity recognition from failing due to term segmentation errors. Therefore, medical-specific word segmenters must be used, and ordinary word segmenters cannot be used.

[0072] For example, in the embodiments of this application, in the terminology text segmentation, "a roundish cavitary shadow visible in the upper lobe of the left lung" is segmented into medical word units such as "upper lobe of the left lung", "visible", "roundish", and "cavitary shadow", to avoid the error of ordinary word segmenters splitting "cavitary shadow" into "cavitary" and "shadow".

[0073] S304. Input the medical vocabulary unit text into the medical corpus model to obtain key medical entities, and identify the key medical entities and label the corresponding entity types through sequence labeling.

[0074] Among them, the boundaries of medical key entities are the starting and ending positions of words or phrases with medical significance; the medical corpus model is obtained by training the Transformer model with the boundaries of medical key entities and medical key entities in the medical vocabulary unit text.

[0075] In some implementations, pre-trained medical domain Transformer models, such as ClinicalBERT and BioBERT, are selected. ClinicalBERT, also known as the clinical BERT model, is a BERT variant optimized for clinical electronic health records, improving performance on clinical NLP tasks through training with medical terminology and clinical text. BioBERT, also known as the biomedical BERT model, is a BERT variant geared towards biomedical research literature, trained with PubMed literature and biomedical knowledge bases, focusing on solving biomedical text understanding problems. According to commonly used annotation rules in medical NER, such as the BIO or BIOES system, different types of key medical entities are labeled. The pre-trained model is fine-tuned using labeled tuberculosis medical corpus, enabling it to learn to recognize lexical units belonging to key medical entities and their corresponding entity types. The text of the medical lexical units to be processed is input into the fine-tuned Transformer model, which outputs corresponding sequence labels for each lexical unit. By filtering based on these labels, complete key medical entities can be obtained and their corresponding entity types matched.

[0076] It should be noted that key medical entities refer to words or phrases with medical significance extracted from medical vocabulary unit text; the labeled entities can be accurately associated with the corresponding regions in the image, making cross-modal feature fusion more in line with medical logic.

[0077] For example, in this embodiment of the application, the terminology text is: "A circular cavitary shadow is visible in the upper lobe of the left lung, with clear boundaries and a diameter of approximately 2.0 cm." The terminology text is split into medical vocabulary unit text using a medical-specific word segmenter: "left upper lobe of the lung," "visible," "circular," "cavitary shadow," "clear boundaries," "approximately," and "2.0 cm in diameter." The medical vocabulary unit text is input into a medical corpus model, and sequence annotation is performed using the BIO annotation system to identify key medical entities and entity types, as shown in Table 1. Table 1

[0078] S305. Organize entity types into structured text data according to a preset format.

[0079] Structured text data refers to structured data that can be directly parsed by computers.

[0080] It should be noted that key medical entities and entity types are isolated pieces of information, lacking logical organization, and cannot be directly understood by computers. Furthermore, structured text data can improve the logic of cross-modal fusion and facilitate data storage and reuse. Therefore, it is necessary to organize entity types according to a preset format to generate structured text data.

[0081] Based on the above technical solutions, medical corpus models and sequence annotations can more automatically and accurately identify key medical entities and entity types, improving the consistency and professionalism of entity information extraction; structured text data improves the automation, efficiency and accuracy of the entire tuberculosis screening process.

[0082] In one possible implementation, combining Figure 2 ,like Figure 4 As shown, after S203, the tuberculosis screening method based on a large model of medical imaging information provided in this application embodiment further includes the following S401 to S403: S401, Segment the standardized image I into... N indivual P×P Size of the tile.

[0083] in, P The length of each tile, , , H It is the height of the image, W It is the width of the image.

[0084] It should be noted that the standardized image size is too large to be directly input into the model. Therefore, the standardized image is split into patches so that the model can extract local detailed features patch by patch.

[0085] S402. Convert the pixel matrix of each patch into a one-dimensional vector. Through the projection matrix E Mapping all one-dimensional vectors of the map tiles to high-dimensional feature vectors And concatenate the projection vectors of all tiles to form a classification token. and the addition of standard position coding The sequence format is obtained. .

[0086] Wherein, the high-dimensional feature vector is ( The sequence format is , It is a learnable classification token. It is the first i The vector formed by the projection of each tile. i ∈[1,N].

[0087] In some implementations, the length of the tile P The selection can be adaptively adjusted based on the type of standardized image: if standardized image I is a chest X-ray, then select... P =16, because the resolution of a standardized chest X-ray image is typically 512. 512 or 1024 1024, therefore the number of pixels per tile is 16. 16=256 would be more suitable for the feature extraction efficiency of the ViT model; if the standardized image I is a chest CT image, then select... P =8, because the resolution of standardized chest CT images is typically 256. 256 or 512 512, therefore the number of pixels per tile is 8. 8=64 allows for more precise capture of subtle lesion textures in CT scans. Simultaneously, if the height of the standardized image... H or width W no P If the height is an integer multiple of the image height, then first perform zero-value padding on the edge regions of the image: pad the image height to... P The smallest integer multiple of the value, such as when H=500 and P=16, padding to 512; the width is similarly padded with zeros before proceeding. P × P The size is divided into N Each tile is arranged in a spatial order from left to right and from top to bottom to match the spatial correspondence of subsequent positional encoding.

[0088] It should be noted that the one-dimensional pixel vectors of the image patch lack semantic information. By mapping them to a higher-dimensional space through a projection matrix, richer visual textures, lesion outlines and other features can be captured. Position encoding enables the model to perceive the spatial location of the image patch in the standardized image.

[0089] For example, suppose the standardized chest X-ray image to be processed has a resolution of 512. A 512-pixel grayscale image with 1 channel; the image is divided into P=16 segments. If there are 16 tiles, then the number of tiles N = (512) 512) / (16 16) = 1024.

[0090] The first step is to divide each of the 16 16 A pixel matrix of 1 is converted into a one-dimensional vector. ,in i ∈[1,1024]; the number of pixels in a single tile is 16. 16 1 = 256, therefore, The dimension is 256.

[0091] The second step is to map the 256-dimensional one-dimensional pixel vector to a 768-dimensional high-dimensional feature space. Therefore, a projection matrix is ​​set. E The dimension is 256×768; the high-dimensional feature vector of each patch is =(256-dimensional vector) (256×768 matrix) = 768-dimensional vector; the set of high-dimensional feature vectors for all tiles is: ( Its dimensions are 1024×768.

[0092] The third step is to introduce a learnable classification token with 768 dimensions. ,Will By concatenating these features at the beginning of all the high-dimensional feature vectors of the tiles, we obtain the concatenated feature set. Its dimensions are (1+1024)×768=1025×768.

[0093] Step 4, Standard Position Encoding The dimension is set to 1025×768 to obtain the sequence format. , The dimensions are 1025×768.

[0094] S403, Format the sequence Input ViT encoder, output global visual feature vector and local visual feature sequences , global visual feature vector and local visual feature sequences Integrate into an image feature sequence.

[0095] The ViT encoder consists of an L-layer multi-head self-attention MSA, a multilayer perceptron MLP, and a layer normalization layer (LayerNorm). The global visual feature vector output by the L-th layer is... The local visual feature sequence corresponding to the 1st to Nth patches output by the Lth layer is: .

[0096] In some implementations, multi-head self-attention The formula for calculation is: ; Where MSA() is the multi-head self-attention function, and LayerNorm() is the layer normalization function. This is the feature vector for each position in the matrix that incorporates information related to other positions. For the first The feature matrix of the layer after processing by the MSA sublayer and the MLP sublayer, and each element in the sequence corresponds to a row in the feature matrix. The feature information of this element is all the values ​​in this row.

[0097] In some implementations, multilayer perceptrons The formula for calculation is: ; Where LayerNorm() represents the layer normalization function, and MLP() represents the multilayer perceptron function. The feature vector representing each position in the matrix has already incorporated the correlation information from other positions in the sequence; It is the first Feature representation after layer processing.

[0098] It should be noted that the global visual feature vector and the local visual feature vector preserve the spatial structure information of the image, which is used for subsequent localization and segmentation tasks.

[0099] Based on the above technical solutions, the semantic expression and spatial correlation of image features are enhanced by high-dimensional projection matrix, classification token and position encoding; the integrated image feature sequence is in a uniform vector form, which can be directly fused with the features of subsequent structured text data across modalities, avoiding the connection cost caused by non-standardized features.

[0100] In one possible implementation, combining Figure 2 ,like Figure 5 As shown, following S204, the tuberculosis screening method based on a large medical imaging information model provided in this application embodiment further includes the following S501 to S504: S501. In some implementations, the global visual feature vector is... With global text feature vector The combined features are obtained by splicing. The input is then fed into a fully connected layer for feature mapping to output a two-dimensional vector. The probability distribution of the two-dimensional vector is calculated using the Softmax function to obtain the positive / negative probability of tuberculosis.

[0101] The formula for calculating the probability distribution is: P cls = softmax ( FC ( )).

[0102] For example, taking the features of chest X-ray images and electronic reports as an example: the ViT encoder outputs a 768-dimensional global visual feature vector. ,like = It contains 768 elements; ClinicalBERT outputs a 768-dimensional global text feature vector. ,like =[0.18,0.24,...,0.12], a total of 768 elements; and By concatenating along the dimensions, we obtain [ Its dimension is 768 + 768 = 1536. The 1536-dimensional combined features are input into a fully connected layer, outputting a two-dimensional vector. For example, if the output is [3.2, -1.8], then the two elements correspond to the original positive and negative scores, respectively. Normalizing the two-dimensional vector using the Softmax function yields: P cls = softmax ([3.2,-1.8])=[ ] Therefore, the probability of identifying the chest X-ray image as pulmonary tuberculosis is 98%, and the probability of identifying it as pulmonary tuberculosis is 2%.

[0103] S502, Fuse local feature sequences Reconstructing the original image to its corresponding two-dimensional spatial layout Using a decoder F 2D Layer-by-layer upsampling to the original image resolution is performed, and a lesion segmentation mask is generated using the Sigmoid activation function. .

[0104] It should be noted that the decoder used is a lightweight U-Net or FPN decoder; the multi-channel feature map obtained by upsampling layer by layer of the decoder is converted into a single-channel lesion probability map through the last 1×1 1×1 convolution, which is used to adapt to the pixel-level segmentation requirements of pulmonary tuberculosis lesions.

[0105] It should also be noted that the output of the last convolutional layer is an unconstrained real number, which cannot directly represent probability. The Sigmoid function is used to map the output value to between 0 and 1, so that the output value has the semantics of probability. The output value of the Sigmoid function corresponds exactly to the lesion probability of the corresponding pixel.

[0106] For example, if the Sigmoid function output value of a pixel is 0.9, it means that the probability of this pixel being a lesion is 90%; if the output value is 0.1, it means that the probability of this pixel being normal tissue is higher.

[0107] S503. Calculate the gradient of the positive category in the tuberculosis positive / negative classification task based on the Grad-CAM algorithm, and apply it to the fused local fusion feature sequence. Gradient-weighted summation is performed to obtain a coarse heatmap. This coarse heatmap is then upsampled to the original image size using bilinear interpolation to obtain a class activation heatmap. .

[0108] In some implementations, the target feature layer for gradient calculation is selected from the locally fused feature sequence after cross-modal attention output. The corresponding feature layer; the average value of the calculated gradient is taken along the feature dimension to obtain the attention weight of each patch, so as to reduce the interference of gradient noise; at the same time, the coarse heatmap is filtered by low weight threshold and then bilinear interpolation upsampling is performed to avoid irrelevant background areas from scattering the heatmap's focus on lesions.

[0109] It should be noted that the heatmap is activated by category. It can intuitively display the area on which the model's decisions are based.

[0110] It should also be noted that bilinear interpolation upsampling is a technique for smoothing upscaled images. It is often used to enlarge low-resolution images to high-resolution ones. The core of it is to fill new pixels by calculating the weighted average of the four surrounding known pixels, avoiding the jaggedness of ordinary stretching and enlargement, and making the enlarged result more natural. For example, coarse heatmaps are upsampled to the original image size using bilinear interpolation, making the thermal distribution of lesion areas smoother and more natural.

[0111] S504, Based on the lesion segmentation mask The corresponding lesion location, size, and morphology information are then fused with the global feature vector. After integration, the data is input into an autoregressive decoder to generate a structured diagnostic report (RR) that matches the lesion information.

[0112] The structured diagnostic report (RR) includes information on the location, size, morphology, and diagnostic conclusion of the lesion.

[0113] In some implementations, the location, size, and morphology of the lesion are first encoded into a fixed-dimensional semantic vector, and then this vector is fused with a global feature vector. The vectors are concatenated and used as the conditional input vector for the autoregressive decoder.

[0114] It should be noted that each layer of the decoder focuses on the fused features through a cross-attention mechanism to ensure that the tokens generated at each step are strongly correlated with the actual information of the lesion.

[0115] For example, in autoregressive generation, the decoder starts with the [BOS] token, and at each step, it uses the token and condition vector generated in the previous step as input to predict the next token, until the [EOS] token is generated, thereby outputting the generated structured diagnostic report RR, such as: A cavity of about 2.0 cm with blurred edges is seen in the upper lobe of the left lung, which is considered to be active pulmonary tuberculosis.

[0116] Based on the above technical solution, the cross-modal features obtained after fusion are used to perform classification, localization and report generation in parallel. By sharing representations, positive transfer between tasks is achieved, providing interpretable diagnostic basis.

[0117] The above primarily describes the solutions of the embodiments of this application from the perspective of device implementation. It is understood that each device, such as a tuberculosis screening device based on a large model of medical image information, includes at least one of the hardware structures and software modules corresponding to each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] This application embodiment can divide the tuberculosis screening device based on a large medical image information model into functional units according to the above method example. For example, each function can be divided into separate functional units, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or software functional units. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0119] In implementation, each step of the method provided in this embodiment can be completed by integrated logic circuits in the processor or by instructions in software form. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0120] The processor in this application may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller (MCU), or an artificial intelligence processor, etc., which are various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor may be a separate semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it may be integrated with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits) to form a System-on-a-Chip (SoC), or it may be integrated as a built-in processor within an ASIC. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0121] The memory in the embodiments of this application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; or electrically erasable programmable-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto.

[0122] This application also provides a computer-readable storage medium including instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0123] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0124] This application also provides a chip including a processor and an interface circuit. The interface circuit is coupled to the processor. The processor is used to run computer programs or instructions to implement the above-described method. The interface circuit is used to communicate with other modules outside the chip.

[0125] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0126] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0127] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A method for screening pulmonary tuberculosis based on a large model of medical imaging information, characterized in that, include: Acquire chest medical images and preprocess them to obtain standardized images; After obtaining chest reports and test data and performing data preprocessing, structured text data is obtained through medical entity recognition. Feature sequences of standardized image data and structured text data are extracted separately and then fused using a cross-modal attention algorithm to generate a fused feature sequence. The feature sequence of the standardized image data is extracted using a pre-trained ViT model, and the feature sequence of the structured text data is extracted using a pre-trained text Transformer model. The fused feature sequence is input into multiple parallel task heads for parallel processing to obtain individual screening results; wherein, the individual screening results include: tuberculosis classification, lesion localization map and diagnostic report.

2. The method for tuberculosis screening based on a large model of medical imaging information according to claim 1, characterized in that, The methods for obtaining the structured text data include: Extract chest reports and test data, and convert the chest reports and test data into text data; Text noise is removed from the text data to obtain the terminology text; The terminology text is split and cleaned using a medical-specific word segmenter to obtain medical vocabulary unit text; wherein, the medical vocabulary unit text is: the terminology text is split into semantically complete medical vocabulary units; Medical vocabulary unit text is input into a medical corpus model to obtain key medical entities. Through sequence labeling, key medical entities are identified and their corresponding entity types are labeled. The medical corpus model is obtained by training a Transformer model with the boundaries of key medical entities in the medical vocabulary unit text and the key medical entities themselves. The boundaries of key medical entities are the start and end positions of words or phrases with medical significance. Organize entity types into structured text data according to a preset format.

3. The method for tuberculosis screening based on a large model of medical imaging information according to claim 1, characterized in that, The method for obtaining the image feature sequence includes: Segment the standardized image I into N indivual P×P Size of the tile; where, P The length of each tile, ), H It is the height of the image, W It is the width of the image; Convert the pixel matrix of each patch into a one-dimensional vector. Through the projection matrix E Mapping all one-dimensional vectors of the map tiles to high-dimensional feature vectors And concatenate the projection vectors of all tiles to form a classification token. and the addition of standard position coding The sequence format is obtained. The sequence format is as follows: , It is a learnable classification token. It is the first i The vector formed by the projection of each tile. i ∈[1,N]; sequence format Input ViT encoder, output global visual feature vector and local visual feature sequences , global visual feature vector and local visual feature sequences The data is integrated into an image feature sequence; wherein, the ViT encoder consists of an L-layer multi-head self-attention MSA, a multilayer perceptron MLP, and a layer normalization layer (LayerNorm), and the global visual feature vector output by the L-th layer is... The local visual feature sequence corresponding to the 1st to Nth patches output by the Lth layer is: .

4. The method according to claim 3, characterized in that, The multi-head self-attention The formula for calculation is: ; Where MSA() is the multi-head self-attention function, and LayerNorm() is the layer normalization function. This is the feature vector for each position in the matrix that incorporates information related to other positions. For the first The feature matrix of the layer after processing by the MSA sublayer and the MLP sublayer.

5. The method according to claim 3, characterized in that, The multilayer sensor The formula for calculation is: ; Where LayerNorm() is the layer normalization function, and MLP() is the multilayer perceptron function. This is the feature vector for each position in the matrix that incorporates information related to other positions. For the first Feature representation after layer processing.

6. The method for tuberculosis screening based on a large medical imaging information model according to claim 1, characterized in that, The methods for obtaining the text feature sequence include: The structured text data is converted into a token sequence using a medical-specific word segmenter. The token sequence is then input into a Transformer model pre-trained on a medical corpus, and the output is a set of semantic feature vectors corresponding to each token in the last layer. The output corresponding to the [CLS] token in the semantic feature vector set is used as the global text feature vector. The output of all word tokens in the semantic feature vector set is used as the local text feature sequence. , global text feature vector and local text feature sequences The features are integrated into a text feature sequence; where the [CLS] token is a fixed starting token preset by the model.

7. The method for tuberculosis screening based on a large medical imaging information model according to claim 1, characterized in that, The method for obtaining the fused feature sequence includes: An attention mechanism is used to calculate the correlation between each tile and all medical vocabulary units, resulting in the initial output feature sequence of the attention mechanism. Perform a dimensionality transformation on the initial output feature sequence, and the output dimension is the same as... Consistent fusion of local fusion feature sequences ; Based on [CLS] token or Pooling is performed to obtain the global fused feature vector. Globally fuse feature vectors With local fusion feature sequences The sequences are integrated into a fusion feature sequence.

8. The method for tuberculosis screening based on a large model of medical imaging information according to claim 7, characterized in that, The process of inputting the fused feature sequence into multiple parallel task heads for parallel processing to obtain individual screening results includes: global visual feature vector With global text feature vector The combined features are obtained by splicing. The input is then fed into a fully connected layer for feature mapping, outputting a two-dimensional vector. The probability distribution of this two-dimensional vector is calculated using the Softmax function to obtain the positive / negative probability of tuberculosis. The formula for calculating the probability distribution is as follows: P cls = softmax ( FC ( )); Local fusion feature sequences Reconstructing the original image to its corresponding two-dimensional spatial layout Using a decoder F 2D Layer-by-layer upsampling to the original image resolution is performed, and a lesion segmentation mask is generated using the Sigmoid activation function. ; The gradient of the positive category in the pulmonary tuberculosis positive / negative classification task is calculated based on the Grad-CAM algorithm, and the fused local fusion feature sequence is then analyzed. Gradient-weighted summation is performed to obtain a coarse heatmap. This coarse heatmap is then upsampled to the original image size using bilinear interpolation to obtain a class activation heatmap. ; Based on the lesion segmentation mask The corresponding lesion location, size, and morphology information are then fused with the global feature vector. After integration, the data is input into an autoregressive decoder to generate a structured diagnostic report (RR) that matches the lesion information.

9. The method for tuberculosis screening based on a large model of medical imaging information according to claim 1, characterized in that, The methods for acquiring the standardized images include: Extract the chest medical images; wherein, the chest medical images include: X-ray images or CT images; Standardized images of the chest are obtained by standardizing the medical images. The standardization process includes grayscale normalization, resampling to a uniform size, and data augmentation.

10. A tuberculosis screening system based on a large model of medical imaging information, characterized in that, The system includes: a standardized image module, a structured text data module, a fused feature sequence module, and an individual screening result module; The standardized imaging module is used to acquire chest medical images and perform preprocessing to obtain standardized images; The structured text data module is used to obtain chest reports and test data, perform data preprocessing, and then obtain structured text data through medical entity recognition. The fusion feature sequence module is used to extract feature sequences from standardized image data and structured text data respectively, and then fuse them to generate a fusion feature sequence through a cross-modal attention algorithm. The feature sequence of the standardized image data is extracted by a pre-trained ViT model, and the feature sequence of the structured text data is extracted by a pre-trained text Transformer model. The individual screening result module is used to input the fused feature sequence into multiple parallel task heads for parallel processing to obtain individual screening results; wherein, the individual screening results include: tuberculosis classification, lesion localization map and diagnostic report.

Images