Medical image diagnosis report automatic generation method based on cross-modal global feature fusion Transform

Abstract

The invention provides a method for automatically generating a medical image diagnosis report based on cross-modal global feature fusion Transformer, and relates to the technical field of image generation. Global feature extraction of a medical image is realized by introducing a sliding window encoder SWE and a grid circulation network MRN; meanwhile, by fusing a cross-modal attention mechanism FMCA and a fusion embedding layer FE, the alignment precision of visual features and text features is remarkably improved, a classification value taking strategy is provided for an initial classification threshold according to a screening index and a threshold division coefficient, and an adjusted classification threshold is formed according to the classification value taking strategy, so that the classification accuracy of the visual features and the text features is improved. The adjusted classification threshold is used for performing grouping setting on the medical images and the diagnosis report texts in the serial number set to form screened medical image and report text subsets; according to the innovative method, the recognition capability of the model on complex lesions is enhanced, and an effective solution is provided for medical image diagnosis automation.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a method for automatically generating a medical image diagnosis report based on a cross-modal global feature fusion Transformer. Background Art

[0002] At present, most medical diagnostic reports still rely on manual writing by radiologists, which is not only time-consuming and laborious, but also easily affected by subjective factors, resulting in inconsistent diagnostic results. For this reason, the automated medical image diagnostic report generation technology based on deep learning has gradually attracted attention. In particular, with the successful application of Transformer in the fields of natural language processing and computer vision, its potential in cross-modal learning has also been gradually explored, promoting the development of automatic generation technology of medical image diagnostic reports based on cross-modal feature fusion; a large number of medical image report writing tasks are complex and tedious tasks for doctors. Therefore, the urgent problem of automatically generating true and reliable diagnostic reports for medical images has gradually attracted widespread attention from researchers in the fields of clinical medicine and artificial intelligence;

[0003] Existing medical image diagnostic report generation technologies still face some shortcomings: first, many methods rely only on local features and ignore the global information of medical images, which is crucial for the accurate identification of complex lesions; second, in the process of cross-modal feature fusion, most existing methods only focus on simple linear fusion and fail to fully utilize the deep semantic association between visual features and text features; it is difficult to achieve medical image-level visual feature extraction and cross-modal data generation in the process of automatic generation of medical image diagnostic reports while maintaining low computing resource overhead; in addition, the existing diagnostic report generation technology lacks flexibility and versatility when processing different types of medical images (such as CT and MRI), resulting in insufficient generalization ability of the model;

[0004] The above information disclosed in the above Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0005] The purpose of the present invention is to provide a method for automatically generating medical image diagnosis reports based on cross-modal global feature fusion Transformer to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for automatically generating medical image diagnostic reports based on a cross-modal global feature fusion Transformer is composed of a medical image visual feature pre-extraction network SwinTransformer, a visual feature encoder and a feature fusion decoder, wherein the visual feature encoder includes a sliding window encoder SWE and a grid recurrent network MRN, and the feature fusion encoder includes a fused cross-modal attention mechanism FMCA and a fusion embedding layer FE. The specific steps include:

[0008] Step S1: obtaining a database of medical image diagnostic reports in a medical image storage system, the database containing a number of medical images and diagnostic report texts directly related thereto, numbering each medical image and the diagnostic report text directly related thereto in pairs to form a number set {1, 2, ..., i, ...n} for each pair of data, wherein i represents the index of the i-th pair of medical image diagnostic report data, and n represents the total number of pairs of medical image diagnostic reports;

[0009] Step S2: Mark the medical image as I and mark the diagnostic report text corresponding to the medical image as X t , where t represents the length of the report text sequence, and features are extracted from the medical images and diagnostic report texts in the number set {1, 2, …, i, …n} to obtain visual feature matrices and text feature vectors;

[0010] Step S3: Quantify the extracted visual feature matrix and text feature vector and define the quantization index Q i It is a comprehensive evaluation index obtained by quantizing the visual feature matrix and text feature vector; according to the quantization index Q i , calculate the screening index S for each pair of medical images and their corresponding diagnostic report text i , set the screening index S i The initial classification threshold of is τ, which is used to group and sort the number set;

[0011] Step S4: obtaining the image quality score of the medical image and the complexity score of the diagnosis report text in the i-th pair of data, and calculating the average value of the image quality score and the complexity score in the current number set;

[0012] Step S5: receiving the average value of the image quality score and the average value of the complexity score in the number set {1, 2, ..., i, ...n}, and performing analysis and processing to generate a threshold partition coefficient, where the threshold partition coefficient is used to provide a classification value strategy for the initial classification threshold;

[0013] Step S6: obtaining a classification value strategy and an initial classification threshold τ, and forming an adjusted classification threshold according to the classification value strategy, wherein the adjusted classification threshold is used to group the medical images and diagnostic report texts in the numbering set to form a subset of medical images and report texts that pass the screening;

[0014] Step S7: Use the pre-trained feature extraction network SwinTransformer to preliminarily extract the medical images and report text subsets grouped in step S6, and obtain the medical image visual feature set V = {v1, v2, ..., v N}, where N is the number of extracted grid features;

[0015] Step S8: The extracted original visual features are dynamically divided into windows using a sliding window encoder SWE;

[0016] Step S9: using a grid recurrent network MRN to average pool the inter-layer information between the sliding window encoders and extract the global information of the medical image as an image-level visual representation;

[0017] Step S10: According to the temporal order, the visual feature representation and global information are sequentially aligned with the diagnosis report text using the feature fusion decoder FFD, and the pathological data and inherent clinical language patterns in the text embedding are learned based on different types of visual features;

[0018] Step S11: Generate the diagnostic report text corresponding to the input image word by word through the learned visual feature mapping, so that the trained cross-modal Transformer model continuously minimizes the loss function between the text modality and the visual features during the generation process to optimize the generation result until the network converges. Finally, the medical image data to be tested is input into the trained cross-modal Transformer model to generate a medical image diagnostic report.

[0019] Further explanation: the image quality score of the i-th medical image is marked as And the corresponding complexity score is marked as

[0020] Calculate the image quality score of the medical image in the i-th pair of data The score will be quantified based on the contrast and noise standard deviation of the image, and the specific formula is as follows:

[0021]

[0022] Among them, C i For image I i Contrast ratio; N i For image I iThe standard deviation of the noise; η1 is a small constant, 0.01≤η1≤0.09, to avoid the situation where the denominator is zero;

[0023] The Laplace operator is used to calculate the contrast of the image. The formula is:

[0024]

[0025] in, represents the second-order derivative of the image, (x, y) represents the pixel coordinates in the image;

[0026] Calculate the noise standard deviation N of the medical image in the i-th pair of data i , by selecting the standard deviation of pixel values ​​in the local area for estimation, the local area selects the center position of the image, and the calculation formula is as follows:

[0027]

[0028] Where M represents the number of pixels in the selected local area, I i ′(k′) represents the normalized image value of the kth pixel in the local area, Represents the average value of pixel values ​​in the local area;

[0029] Calculate the quality scores of all images in the numbered set {1,2,…,i,…n} The average value of The calculation formula is as follows:

[0030]

[0031] Determining the complexity score The feature values ​​included are term density, syntactic complexity, information entropy, sentiment analysis score and text readability score. The score range of each feature value is set to be an integer from 1 to 10, and the larger the score value, the higher the corresponding feature complexity. Any feature value in the diagnosis report text is used as T j,i Denote, j∈{1,2,3,4,5}, and label the term density as T 1,i , the syntactic complexity is marked as T 2,i , information entropy is marked as T 3,i , the sentiment analysis score is marked as T 4,i , the text readability score is marked as T 5,i ; The calculation formula is as follows:

[0032]

[0033] Among them, a1, a2, a3, a4, and a5 represent the weights of the corresponding eigenvalues, and

[0034] Compute all complexity scores in the numbered set {1,2,…,i,…n} The average value of The calculation formula is as follows:

[0035]

[0036] Further, the image quality scores in the receiving number set {1, 2, ..., i, ...n} are The average And complexity score The average And analyze and process to generate threshold division coefficients. The threshold division coefficients are used to provide classification value strategies for the initial classification thresholds, including:

[0037] The threshold division coefficient is defined as κ, and the calculation formula is:

[0038]

[0039] Among them, κ is the threshold division coefficient, is the average of the image quality scores in the numbered set {1, 2, …, i, …n}, representing the clarity and noise ratio of the image; is the average of the complexity scores in the numbered set {1,2,…,i,…n}, indicating the density of medical terms and grammatical complexity in the text;

[0040] γ is a tuning parameter used to control the interaction strength between the image quality score and the text complexity score, γ>0; if the image quality score Increase or text complexity score decreases, the threshold division coefficient κ will increase; λ is the weight parameter, which adjusts the influence of the sum of squares of eigenvectors on the overall calculation, 0<λ<1; T j,i is the jth eigenvalue of the diagnosis report text in the i-th pair of data; 5 is the feature number of the diagnosis report text, indicating the dimension of the extracted text feature vector; is the variance of the visual feature matrix in the i-th pair of data, reflecting the distribution of image features; the smaller the variance, the more concentrated the image features, and the larger the calculated threshold division coefficient κ; i is a positive value, 0.01≤ε i ≤0.03, ε i Prevent the denominator from being zero;

[0041] The effective range of the threshold division coefficient κ is set between [0.1, 0.75]. Different values ​​within the range will provide the following different classification value strategies for the initial classification threshold τ;

[0042] When 0.1≤κ<0.25, it means that the image quality and text complexity are balanced or the image features are relatively dispersed. Improving τ can reduce the impact of low-quality data. The τ adjustment formula is as follows:

[0043]

[0044] Among them, τ1′ is the adjusted τ;

[0045] When 0.25≤κ<0.56, the image quality matches the text complexity, and maintaining the current τ is conducive to accurate screening;

[0046] When κ≥0.56, the image quality is high or the text complexity is low, and the image features are concentrated. Reducing τ can retain high-quality data for analysis. The τ adjustment formula is as follows:

[0047]

[0048] Among them, τ2′ is the adjusted τ;

[0049] According to the adjusted classification thresholds τ1′, τ2′, τ1′>τ2′, the medical images and diagnostic report texts in the numbered set are grouped and set. The specific definition formula is:

[0050]

[0051] Among them, for {(I i ,X t,i )|S i ≥τ1′}, which means that the number set {1,2,…,i,…n} that meets S i Each pair of data with ≥τ1′ is taken as the first classification group;

[0052] For {(I i ,X t,i )|τ1′ i <τ2′}, which means to convert all the numbers in the set {1,2,…,i,…n} that match τ1′ i Each pair of data <τ2′ is taken as the second classification group;

[0053] For {(I i ,X t,i )|S i <τ2′}, which means that all the numbers in the set {1,2,…,i,…n} that match S i Each pair of data with <τ2′ is taken as the third classification group.

[0054] ​​Compared with the prior art, the beneficial effects of the present invention are as follows: by introducing the sliding window encoder SWE and the grid recurrent network MRN, the global feature extraction of medical images is realized; at the same time, by fusing the cross-modal attention mechanism FMCA and the fusion embedding layer FE, the alignment accuracy of visual features and text features is significantly improved, and a classification value strategy is provided for the initial classification threshold according to the screening index and the threshold division coefficient, and an adjusted classification threshold is formed according to the classification value strategy, and the adjusted classification threshold is used to group and set the medical images and diagnostic report texts in the numbering set to form a subset of medical images and report texts that pass the screening; in the process of automatically generating medical image diagnostic reports, medical image-level visual feature extraction and cross-modal data generation are realized, while maintaining a low computing resource overhead;

[0055] This innovative approach not only enhances the model's ability to recognize complex lesions, but also improves the accuracy and consistency of generated diagnostic reports, providing a more effective solution for automated medical image diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the overall method flow of the present invention. DETAILED DESCRIPTION

[0057] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0058] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0059] Embodiment 1:

[0060] See also Figure 1 , the present invention provides a technical solution:

[0061] The automatic generation method of medical image diagnosis report based on cross-modal global feature fusion Transformer is composed of a medical image visual feature pre-extraction network SwinTransformer, a visual feature encoder and a feature fusion decoder, wherein the visual feature encoder includes a sliding window encoder SWE and a grid recurrent network MRN, and the feature fusion encoder includes a fusion cross-modal attention mechanism FMCA and a fusion embedding layer FE, including the following steps:

[0062] Step S1: obtaining a database of medical image diagnostic reports in a medical image storage system, the database containing a number of medical images and diagnostic report texts directly related thereto, numbering each medical image and the diagnostic report text directly related thereto in pairs to form a number set {1, 2, ..., i, ...n} for each pair of data, wherein i represents the index of the i-th pair of medical image diagnostic report data, and n represents the total number of pairs of medical image diagnostic reports;

[0063] Step S2: Mark the medical image as I and mark the diagnostic report text corresponding to the medical image as X t , where t represents the length of the report text sequence, feature extraction is performed on the medical images and diagnostic report texts in the number set {1, 2, …, i, …n}, and the visual feature matrix and text feature vector are obtained;

[0064] Step S3: Quantify the extracted visual feature matrix and text feature vector and define the quantization index Q i It is a comprehensive evaluation index obtained by quantizing the visual feature matrix and text feature vector; according to the quantization index Q i , calculate the screening index S for each pair of medical images and their corresponding diagnostic report text i , set the screening index S i The initial classification threshold of is τ, which is used to group and sort the number sets;

[0065] Step S4: obtaining the image quality score of the medical image and the complexity score of the diagnosis report text in the i-th pair of data, and calculating the average value of the image quality score and the complexity score in the current number set;

[0066] Step S5: receiving the average value of the image quality score and the average value of the complexity score in the number set {1, 2, ..., i, ...n}, and performing analysis and processing to generate a threshold partition coefficient, where the threshold partition coefficient is used to provide a classification value strategy for the initial classification threshold;

[0067] Step S6: obtaining a classification value strategy and an initial classification threshold τ, and forming an adjusted classification threshold according to the classification value strategy, wherein the adjusted classification threshold is used to group the medical images and diagnostic report texts in the numbering set to form a subset of medical images and report texts that pass the screening;

[0068] Step S7: Use the pre-trained feature extraction network SwinTransformer to preliminarily extract the medical images and report text subsets grouped in step S6, and obtain the medical image visual feature set V = {v1, v2, ..., v N}, where N is the number of extracted grid features;

[0069] Step S8: The extracted original visual features are dynamically divided into windows using a sliding window encoder SWE; ensuring that the edge information between different sub-images is omitted and capturing multi-scale visual features to extract the rich pathological information V contained in the medical image L , while reducing the model calculation pressure;

[0070] Step S9: using a grid recurrent network MRN to average pool the inter-layer information between the sliding window encoders and extract the global information of the medical image as an image-level visual representation;

[0071] Step S10: According to the temporal order, the visual feature representation and global information are sequentially aligned with the diagnosis report text using the feature fusion decoder FFD, and the pathological data and inherent clinical language patterns in the text embedding are learned based on different types of visual features;

[0072] Step S11: Generate the diagnostic report text corresponding to the input image word by word through the learned visual feature mapping, so that the trained cross-modal Transformer model continuously minimizes the loss function between the text modality and the visual features during the generation process to optimize the generation result until the network converges. Finally, the medical image data to be tested is input into the trained cross-modal Transformer model to generate a medical image diagnostic report.

[0073] It is further stated that the database should include multiple disease types and multiple patient samples to ensure data diversity;

[0074] In the data structure, each pair of data is represented as (I i ,X t,i ), where I i is the i-th medical image, X t,i is the diagnostic report text corresponding to the i-th medical image;

[0075] Feature extraction is performed on the medical images and diagnostic report texts in the number set {1, 2, ..., i, ...n}, specifically including:

[0076] For medical images, a convolutional neural network (CNN) is used to extract visual features and generate a visual feature matrix V i ; For the diagnostic report text, natural language processing (NLP) technology is used to process the text and generate a text feature vector T i ; The mathematical formula is as follows:

[0077]

[0078] Wherein, f and g represent the functions for feature extraction of medical images and diagnosis report text respectively;

[0079] Define the quantitative indicator Q i The calculation formula is as follows:

[0080] Q i =αV i +βT i

[0081] Among them, α and β are weight coefficients, The specific values ​​of α and β are adjusted by an expert group based on experimental data according to the data characteristics in order to balance the influence of visual and text features;

[0082] Define the screening index S i The calculation formula is as follows:

[0083]

[0084] Among them, S i Used to calculate the ratio of the i-th pair of data to ensure the screening index S i The value range is between (0,1); it indicates the relative importance of the pair of data in the entire data set;

[0085] Further explanation: the image quality score of the i-th medical image is marked as And the corresponding complexity score is marked as

[0086] Calculate the image quality score of the medical image in the i-th pair of data The score will be quantified based on the contrast and noise standard deviation of the image, and the specific formula is as follows:

[0087]

[0088] Among them, C i For image I i Contrast ratio, N i For image I iThe standard deviation of the noise; η1 is a small constant, 0.01≤η1≤0.09, to avoid the situation where the denominator is zero;

[0089] The Laplace operator is used to calculate the contrast of the image. The formula is:

[0090]

[0091] in, represents the second-order derivative of the image, (x, y) represents the pixel coordinates in the image;

[0092] Calculate the noise standard deviation N of the medical image in the i-th pair of data i , by selecting the standard deviation of pixel values ​​in the local area for estimation, the local area selects the center of the image, and the local area can also be adjusted by the expert group according to the experimental data. The formula is as follows:

[0093]

[0094] Where M represents the number of pixels in the selected local area, I i ′(k′) represents the normalized image value of the kth pixel in the local area, Represents the average value of pixel values ​​in the local area;

[0095] Calculate the quality scores of all images in the numbered set {1,2,…,i,…n} The average value of The calculation formula is as follows:

[0096]

[0097] Determining the complexity score The feature values ​​included are term density, syntactic complexity, information entropy, sentiment analysis score and text readability score. The score range of each feature value is set to be an integer from 1 to 10, and the larger the score value, the higher the corresponding feature complexity. Any feature value in the diagnosis report text is used as T j,i Denote, j∈{1,2,3,4,5}, and label the term density as T 1,i , the syntactic complexity is marked as T 2,i , information entropy is marked as T 3,i , the sentiment analysis score is marked as T 4,i , the text readability score is marked as T 5,i ; The calculation formula is as follows:

[0098]

[0099] Among them, a1, a2, a3, a4, and a5 represent the weights of the corresponding eigenvalues, and

[0100] Compute all complexity scores in the numbered set {1,2,…,i,…n} The average value of The calculation formula is as follows:

[0101]

[0102] 1.1) According to the frequency of use of professional terms, the scoring rules for term density T1,i are as follows:

[0103] 1-2 points: term density is 0%-20%;

[0104] 3-4 points: term density is 21%-40%;

[0105] 5-6 points: term density is 41%-60%;

[0106] 7-8 points: term density is 61%-80%;

[0107] 9-10 points: term density is 81%-100%;

[0108] 1.2) Score according to the complexity of the sentence structure; the scoring rules for syntactic complexity T2,i are as follows:

[0109] 1-2 points: syntactic complexity is 0%-20%;

[0110] 3-4 points: syntactic complexity is 21%-40%;

[0111] 5-6 points: syntactic complexity is 41%-60%;

[0112] 7-8 points: syntactic complexity is 61%-80%;

[0113] 9-10 points: syntactic complexity is 81%-100%;

[0114] 1.3) According to the diversity and distribution of text information, the scoring rules for information entropy T3,i are as follows:

[0115] 1-2 points: Information entropy is 0%-20%, with high content concentration and many repetitions;

[0116] 3-4 points: Information entropy is 21%-40%, and information distribution is relatively concentrated;

[0117] 5-6 points: Information entropy is 41%-60%, and information distribution is relatively balanced;

[0118] 7-8 points: Information entropy is 61%-80%, with diverse information distribution and rich content;

[0119] 9-10 points: Information entropy is 81%-100%, the information is highly diverse and the content is extremely rich;

[0120] 1.4) According to the intensity and tendency of the text sentiment expression, the sentiment analysis score T4,i is scored as follows:

[0121] 1-2 points: negative emotion intensity is 81%-100%;

[0122] 3-4 points: negative emotion intensity is 61%-80%;

[0123] 5-6 points: neutral or mixed emotions, with a negative or positive intensity of 41%-60%;

[0124] 7-8 points: positive emotion intensity is 61%-80%;

[0125] 9-10 points: positive emotion intensity is 81%-100%;

[0126] 1.5) According to the readability score of the text, the scoring rules for the text readability score T5,i are as follows:

[0127] 1-2 points: readability is 0%-20%, extremely difficult to read;

[0128] 3-4 points: readability is 21%-40%, which is difficult to read;

[0129] 5-6 points: readability is 41%-60%, and the average reader can understand it;

[0130] 7-8 points: readability is 61%-80%, most readers can easily understand;

[0131] 9-10 points: readability is 81%-100%, very easy to read, simple and clear;

[0132] Further explanation: the image quality scores in the receiving number set {1, 2, ..., i, ...n} The average And complexity score The average And analyze and process to generate threshold division coefficients. The threshold division coefficients are used to provide classification value strategies for the initial classification thresholds, including:

[0133] The threshold division coefficient is defined as κ, and the calculation formula is:

[0134]

[0135] Among them, κ is the threshold division coefficient, is the average of the image quality scores in the numbered set {1, 2, …, i, …n}, representing the clarity and noise ratio of the image; is the average of the complexity scores in the numbered set {1,2,…,i,…n}, indicating the density of medical terms and grammatical complexity in the text;

[0136] γ is a tuning parameter used to control the interaction strength between the image quality score and the text complexity score, γ>0; if the image quality score Increase or text complexity score decreases, the threshold division coefficient κ will increase; λ is the weight parameter, which adjusts the influence of the sum of squares of eigenvectors on the overall calculation, 0<λ<1; T j,i is the jth eigenvalue of the diagnosis report text in the i-th pair of data; 5 is the feature number of the diagnosis report text, indicating the dimension of the extracted text feature vector; is the variance of the visual feature matrix in the i-th pair of data, reflecting the distribution of image features; the smaller the variance, the more concentrated the image features, and the larger the calculated threshold division coefficient κ; i is a positive value, 0.01≤ε i ≤0.03, ε i Prevent the denominator from being zero;

[0137] The effective range of the threshold division coefficient κ is set between [0.1, 0.75]. When the value of κ is in the interval [0.1, 0.75], no adjustment strategy is provided for the initial classification threshold τ; the specific adjustment depends on the scoring range and feature distribution of the actual data; different values ​​within the range will provide the following different classification value strategies for the initial classification threshold τ;

[0138] When 0.1≤κ<0.25, it means that the image quality and text complexity are balanced or the image features are relatively dispersed. Improving τ can reduce the impact of low-quality data. The τ adjustment formula is as follows:

[0139]

[0140] Among them, τ1′ is the adjusted τ;

[0141] When 0.25≤κ<0.56, the image quality matches the text complexity, and maintaining the current τ is conducive to accurate screening;

[0142] When κ≥0.56, the image quality is high or the text complexity is low, and the image features are concentrated. Reducing τ can retain high-quality data for analysis. The τ adjustment formula is as follows:

[0143]

[0144] Among them, τ2′ is the adjusted τ;

[0145] According to the adjusted classification thresholds τ1′, τ2′, τ1′>τ2′, the medical images and diagnostic report texts in the numbered set are grouped and set. The specific definition formula is:

[0146]

[0147] Among them, for {(I i ,X t,i )|S i ≥τ1′}, which means that the number set {1,2,…,i,…n} that meets S i Each pair of data with ≥τ1′ is taken as the first classification group;

[0148] For {(I i ,X t,i )|τ1′ i <τ2′}, which means to convert all the numbers in the set {1,2,…,i,…n} that match τ1′ i Each pair of data <τ2′ is taken as the second classification group;

[0149] For {(I i ,X t,i )|S i <τ2′}, which means that all the numbers in the set {1,2,…,i,…n} that match S i Each pair of data <τ2′ is taken as the third classification group;

[0150] In summary, a subset of medical images and report texts that passed the screening was obtained;

[0151] The screened medical images and report texts are grouped into similar categories, and the data are further analyzed using clustering algorithms (such as K-means or DBSCAN);

[0152] Output the grouping results in the form of structured data for subsequent analysis; the output format is JSON or CSV; to facilitate further processing and visualization of the data;

[0153] Further explanation: the process of extracting preliminary medical image visual features by the pre-trained feature extraction network SwinTransformer is as follows:

[0154] The pre-extraction network divides the given medical image into several sub-images, and then updates these sub-images through the alternating window and sliding window attention mechanisms, so as to efficiently learn the correlation between the sub-images and then construct the grid features of the medical image for subsequent learning, and obtain the following set of medical image visual features:

[0155] ​​

[0156] Among them, V represents the set of initially extracted visual features, v m represents the visual feature of the mth grid feature, I represents the original medical image after grouping in step S6, Swin(·) represents the SwinTransformer feature extractor, and N is the number of extracted grid features; in this embodiment, (N=7×7), D is the embedding dimension of each grid feature;

[0157] It is further explained that the sliding window encoder comprises L layers, each layer is composed of an alternating window attention mechanism W-MSA or a sliding window attention SW-MSA mechanism, which improves the encoding ability of the encoder by integrating pathological features from different scales, while reducing the computational complexity of the multi-head attention mechanism;

[0158] The working process of the sliding window encoder is:

[0159] First, the original grid features are learned by alternating different windows through the sliding window attention mechanism; initially, the original features are divided into several sub-graphs of scale m′; subsequently, the self-attention mechanism is applied to each sub-graph, and the formula is expressed as:

[0160]

[0161] in, Represent the training parameters of the i′th window, w represents the total number of windows, w i′ represents the window subgraph, MSA(·) represents the multi-head self-attention mechanism; W-MSA merges the features of all windows after extracting them, and the formula is expressed as:

[0162] SW-MSA=(Q,K,V)=Merge(w1,...,w w )

[0163] Merge(·) represents the window merging operation, SW-MSA represents the sliding window attention mechanism, and Q, K, and V represent the learnable embedding matrices respectively. Since the window size m′ is smaller than the original image size h or the total number of windows w, the calculation amount of the attention mechanism that presents quadratic growth in the conventional attention mechanism is reduced. In addition, in order to compensate for the information loss caused by data fragmentation due to window division, SW-MSA is used to promote the cross-learning of information in windows at different positions, so that adjacent windows can interact with each other.

[0164] After completing the above steps, the original features retain the extracted pathological information while operating at a lower level of computational complexity. The overall SWE formula is expressed as:

[0165]

[0166]

[0167] in, is the feature representation after the sliding window attention mechanism and layer normalization at layer L, z L-1 It is the output feature representation of the L-1th layer, which plays the role of transmitting information to the next layer in the multi-layer neural network or encoder. L and V L-1 Respectively represent the output between the L layer and the L-1 layer in the encoder; in addition, V 0 =V, is a learnable parameter matrix; LayerNorm(·) represents the normalization operation, and the FeedForward(·) operation consists of a linear layer and a ReLU activation function; the output of the encoder is used in the decoder and MRN for subsequent learning;

[0168] Further explanation: the grid recurrent network module extracts the inter-layer information of the sliding window encoder as the global feature representation through the pooling layer and the gated recurrent unit; the working process of the grid recurrent network is:

[0169] First, the information of each layer in the sliding window encoder is collected through the pooling operation, and the visual features between each layer of information are fused to capture the information between different layers;

[0170] Subsequently, in order to address the limitations of long-term memory and back-propagated gradients while reducing the resulting computational complexity, a gated recurrent unit was adopted as a global feature extractor;

[0171] Further explanation: the feature fusion decoder includes a fusion attention mechanism FMCA and a fusion embedding layer FE; the working process of the feature fusion decoder is:

[0172] The global features are matched with the reported text data using a fusion embedding layer to improve the global reasoning ability of the generative model;

[0173] The image-level visual representation extracted from MRN is used as the original global feature to interact with the text data to capture the global information contained in the medical image; the following calculation formula for the t-th text feature map is obtained:

[0174]

[0175] in, is the final text feature map of the dth layer, which represents the result of fusing the first t-1 text embeddings. FE(·) represents the fusion embedding layer, which is used to combine the global features from the image with the text embedding. W g is a learnable parameter, is the global feature extracted from the medical image, X represents the text embedding, Indicates the first d-1 The first t-1 text embeddings, It is the first d-1 The first t-1 text embeddings of the layer, Concat(·) is a feature concatenation operation, Concat(·) is used to and Taken together, EL(·) is a nonlinear activation function; this embodiment is any one of ReLU and Sigmoid, which processes the concatenated and transformed feature vectors to increase the network's expressive power and nonlinearity, so that the resulting feature map can better capture complex patterns and relationships;

[0176] Embedding visual features and text into X t Fusion, using the visual features V learned by the sliding window encoder in a multi-head cross-modal attention mechanism L As a query, use the output of the masked attention mechanism As a key-value pair, it is expressed as follows:

[0177]

[0178] in, are three trainable parameter matrices, and MCA(·) stands for cross-modal attention mechanism;

[0179] The global features are fused with the output of the cross-modal attention mechanism, and additional jump connections are introduced to protect the fragile information flow during the propagation process. The formula is expressed as:

[0180]

[0181] in, and g L They are the outputs of FMCA and MRN respectively; FMCA stands for “Fusion with Multimodal Cross-Attention”.

[0182] Considering that multi-angle fusion will inevitably lead to an increase in memory usage, a mid-fusion strategy is introduced to reduce redundancy and retain the useful original features observed before the final fusion; f Previously, global features were only fused with text features; then, f Starting from the layer, the global features are no longer learned together with the text features; instead, the FMCA module is responsible for extracting global features at this stage; the above operation formula is expressed as follows:

[0183]

[0184] Among them, l d Indicates the current layer number, l f Indicates the threshold of the number of layers for fusing text features, g L Represents the global information output by MRN, Represents the text features after processing in the previous layer;

[0185] Finally, an activation function feed-forward network is applied to generate the corresponding reported results.

[0186] Further, in step S11, the report is generated word by word through the learned visual feature mapping, aiming to minimize the loss function of text modality and visual features, so as to update the network parameters until the network converges, specifically including:

[0187] In the text mode, we first use the word-level cross entropy loss function to measure the difference between the generated report and the reference report. The formula is:

[0188]

[0189] Among them, i″ is the word index, l represents the number of words, is the true report sequence, p θ (·) represents the probability function, L XE (θ) represents the value of the cross entropy loss function;

[0190] In order to strengthen the model's reasoning optimization process for different modalities, in the visual modality, the medical images in the retrieval dataset and the corresponding paired and unpaired reports are used to enhance the model's cross-modal alignment capability by increasing the difference between visual features and paired and unpaired reports. The formula is expressed as:

[0191] L V =max(dis(z v ,z x )-dis(z v ,z x' )+α′,0)

[0192]

[0193] Among them, z v The visual feature vector representing the medical image, which represents the features extracted from the image for cross-modal alignment, z x Represents the paired text embedding vector, which represents the image z v Features of paired report text, z x' Represents the unpaired text embedding vector, which is expressed with image z vFeatures of unmatched report text; dis(·,·) represents the cross-modal distance evaluation function based on cosine similarity, α′ represents the edge threshold to control the difference between paired and unpaired text of image pairs, cos(·) represents the calculation of cosine similarity, L V Represents the loss function for the visual modality; this loss is used to optimize the cross-modal alignment ability of the model, aiming to increase the difference between visual features and paired and unpaired reports.

[0194] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0195] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0196] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0197] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for automatically generating medical image diagnosis reports based on cross-modal global feature fusion Transformer, characterized in that: It consists of a medical image visual feature pre-extraction network SwinTransformer, a visual feature encoder and a feature fusion decoder. The visual feature encoder includes a sliding window encoder SWE and a grid recurrent network MRN. The feature fusion encoder includes a fused cross-modal attention mechanism FMCA and a fusion embedding layer FE. The specific steps include: Step S1: obtaining a database of medical image diagnostic reports in a medical image storage system, the database containing a number of medical images and diagnostic report texts directly related thereto, numbering each medical image and the diagnostic report text directly related thereto in pairs to form a number set {1, 2, ..., i, ...n} for each pair of data, wherein i represents the index of the i-th pair of medical image diagnostic report data, and n represents the total number of pairs of medical image diagnostic reports; Step S2: Mark the medical image as I and mark the diagnostic report text corresponding to the medical image as X t , where t represents the length of the report text sequence, and features are extracted from the medical images and diagnostic report texts in the number set {1, 2, …, i, …n} to obtain visual feature matrices and text feature vectors; Step S3: Quantify the extracted visual feature matrix and text feature vector and define the quantization index Q i is the comprehensive evaluation index obtained after quantization of the visual feature matrix and the text feature vector; according to the quantization index Qi, the screening index S of each pair of medical images and their corresponding diagnostic report text is calculated i , set the screening index S i The initial classification threshold of is τ, which is used to group and sort the number set; Step S4: obtaining the image quality score of the medical image and the complexity score of the diagnosis report text in the i-th pair of data, and calculating the average value of the image quality score and the complexity score in the current number set; Step S5: receiving the average value of the image quality score and the average value of the complexity score in the number set {1, 2, ..., i, ...n}, and performing analysis and processing to generate a threshold partition coefficient, where the threshold partition coefficient is used to provide a classification value strategy for the initial classification threshold; Step S6: obtaining a classification value strategy and an initial classification threshold τ, and forming an adjusted classification threshold according to the classification value strategy, wherein the adjusted classification threshold is used to group the medical images and diagnostic report texts in the numbering set to form a subset of medical images and report texts that pass the screening; Step S7: Use the pre-trained feature extraction network SwinTransformer to preliminarily extract the medical images and report text subsets grouped in step S6, and obtain the medical image visual feature set V = {v1, v2, ..., v N }, where N is the number of extracted grid features; Step S8: The extracted original visual features are dynamically divided into windows using a sliding window encoder SWE; Step S9: using a grid recurrent network MRN to average pool the inter-layer information between the sliding window encoders and extract the global information of the medical image as an image-level visual representation; Step S10: According to the temporal order, the visual feature representation and global information are sequentially aligned with the diagnosis report text using the feature fusion decoder FFD, and the pathological data and inherent clinical language patterns in the text embedding are learned based on different types of visual features; Step S11: Generate the diagnostic report text corresponding to the input image word by word through the learned visual feature mapping, so that the trained cross-modal Transformer model continuously minimizes the loss function between the text modality and the visual features during the generation process to optimize the generation result until the network converges. Finally, the medical image data to be tested is input into the trained cross-modal Transformer model to generate a medical image diagnostic report.

2. According to claim 1, a method for automatically generating medical image diagnosis reports based on cross-modal global feature fusion Transformer is characterized by: In the data structure, each pair of data is represented as (I i ,X t,i ), where I i is the i-th medical image, X t,i is the diagnostic report text corresponding to the i-th medical image; Feature extraction is performed on the medical images and diagnostic report texts in the number set {1, 2, ..., i, ...n}, specifically including: For medical images, a convolutional neural network is used to extract visual features and generate a visual feature matrix V i ; For the diagnostic report text, natural language processing technology is used to process the text and generate a text feature vector T i ; The mathematical formula is as follows: Wherein, f and g represent the functions for feature extraction of medical images and diagnosis report text respectively; Define the quantitative indicator Q i The calculation formula is as follows: Q i =αV i +βT i Among them, α and β are weight coefficients, Define the screening index S i The calculation formula is as follows: Among them, S i Used to calculate the ratio of the i-th pair of data to ensure the screening index S i The value range is between (0,1).

3. According to claim 2, a method for automatically generating medical image diagnosis reports based on cross-modal global feature fusion Transformer is characterized by: The image quality score of the i-th medical image is marked as And the corresponding complexity score is marked as Calculate the image quality score of the medical image in the i-th pair of data The score will be quantified based on the contrast and noise standard deviation of the image, and the specific formula is as follows: Among them, C i For image I i Contrast ratio; N i For image I i The standard deviation of the noise; η1 is a small constant, 0.01≤η1≤0.09, to avoid the situation where the denominator is zero; The Laplace operator is used to calculate the contrast of the image. The formula is: in, represents the second-order derivative of the image, (x, y) represents the pixel coordinates in the image; Calculate the noise standard deviation N of the medical image in the i-th pair of data i , by selecting the standard deviation of pixel values ​​in the local area for estimation, the local area selects the center position of the image, and the calculation formula is as follows: Where M represents the number of pixels in the selected local area, I′ i (k′) represents the normalized image value of the kth pixel in the local area, Represents the average value of pixel values ​​in the local area; Calculate the quality scores of all images in the numbered set {1,2,…,i,…n} The average value of The calculation formula is as follows: Determining the complexity score The feature values ​​included are term density, syntactic complexity, information entropy, sentiment analysis score and text readability score. The score range of each feature value is set to be an integer from 1 to 10, and the larger the score value, the higher the corresponding feature complexity. Any feature value in the diagnosis report text is used as T j,i Denote, j∈{1,2,3,4,5}, and label the term density as T 1,i , the syntactic complexity is marked as T 2,i , information entropy is marked as T 3,i , the sentiment analysis score is marked as T 4,i , the text readability score is marked as T 5,i ; The calculation formula is as follows: Among them, a1, a2, a3, a4, and a5 represent the weights of the corresponding eigenvalues, and Compute all complexity scores in the numbered set {1,2,…,i,…n} The average value of The calculation formula is as follows:

4. According to claim 3, a method for automatically generating medical image diagnosis reports based on cross-modal global feature fusion Transformer is characterized in that: Receive image quality scores in the set of numbers {1,2,…,i,…n} The average And complexity score The average And analyze and process to generate threshold division coefficients. The threshold division coefficients are used to provide classification value strategies for the initial classification thresholds, including: The threshold division coefficient is defined as κ, and the calculation formula is: Among them, κ is the threshold division coefficient, is the average of the image quality scores in the numbered set {1, 2, …, i, …n}, representing the clarity and noise ratio of the image; is the average of the complexity scores in the numbered set {1,2,…,i,…n}, indicating the density of medical terms and grammatical complexity in the text; γ is a tuning parameter used to control the interaction strength between the image quality score and the text complexity score, γ>0; if the image quality score Increase or text complexity score decreases, the threshold division coefficient κ will increase; λ is the weight parameter, which adjusts the influence of the sum of squares of eigenvectors on the overall calculation, 0<λ<1; T j,i is the jth eigenvalue of the diagnosis report text in the i-th pair of data; 5 is the feature number of the diagnosis report text, indicating the dimension of the extracted text feature vector; is the variance of the visual feature matrix in the i-th pair of data, reflecting the distribution of image features; the smaller the variance, the more concentrated the image features, and the larger the calculated threshold division coefficient κ; i is a positive value, 0.01≤ε i ≤0.03, εi prevents the denominator from being zero; The effective range of the threshold division coefficient κ is set between [0.1, 0.75]. Different values ​​within the range will provide the following different classification value strategies for the initial classification threshold τ; When 0.1≤κ<0.25, it means that the image quality and text complexity are balanced or the image features are relatively dispersed. Improving τ can reduce the impact of low-quality data. The τ adjustment formula is as follows: Among them, τ1′ is the adjusted τ; When 0.25≤κ<0.56, the image quality matches the text complexity, and maintaining the current τ is conducive to accurate screening; When κ≥0.56, the image quality is high or the text complexity is low, and the image features are concentrated. Reducing τ can retain high-quality data for analysis. The τ adjustment formula is as follows: Among them, τ2′ is the adjusted τ; According to the adjusted classification thresholds τ1′, τ2′, τ1′>τ2′, the medical images and diagnostic report texts in the numbered set are grouped and set. The specific definition formula is: Among them, for {(I i ,X t,i )|S i ≥τ1′}, which means that the number set {1,2,…,i,…n} that meets S i Each pair of data with ≥τ1′ is taken as the first classification group; For {(I i ,X t,i )|τ1′ i <τ2′}, which means to convert all the numbers in the set {1,2,…,i,…n} that match τ1′ i Each pair of data <τ2′ is taken as the second classification group;​​ For {(I i ,X t,i )|S i <τ2′}, which means that all the numbers in the set {1,2,…,i,…n} that match S i Each pair of data with <τ2′ is taken as the third classification group.

5. According to claim 4, a method for automatically generating medical image diagnosis reports based on cross-modal global feature fusion Transformer is characterized in that: The process of extracting preliminary medical image visual features by the pre-trained feature extraction network SwinTransformer is as follows: The pre-extraction network divides the given medical image into several sub-images, and then updates these sub-images through the alternating window and sliding window attention mechanisms to construct the grid features of the medical image, and obtain the following set of medical image visual features: Among them, V represents the set of initially extracted visual features, v m represents the visual feature of the mth grid feature, I represents the original medical image after grouping in step S6, and Swin(·) represents the SwinTransformer feature extractor.

6. The method for automatically generating medical image diagnosis reports based on cross-modal global feature fusion Transformer according to claim 5 is characterized in that: The sliding window encoder comprises L layers, each layer consisting of an alternating window attention mechanism W-MSA or a sliding window attention SW-MSA mechanism; The grid recurrent network module extracts the inter-layer information of the sliding window encoder as a global feature representation through a pooling layer and a gated recurrent unit; The feature fusion decoder includes a fusion attention mechanism FMCA and a fusion embedding layer FE; the working process of the feature fusion decoder is: The global features are matched with the reported text data using a fusion embedding layer to improve the global reasoning ability of the generative model; The image-level visual representation extracted from MRN is used as the original global feature to interact with text data to capture the global information contained in medical images; The following calculation formula for the t-th text feature map is obtained: in, is the final text feature map of the dth layer, FE(·) represents the fusion embedding layer, which is used to combine the global features from the image with the text embedding, W g is a learnable parameter, is the global feature extracted from the medical image, X represents the text embedding, Indicates the first d-1 The first t-1 text embeddings of the layer, Concat(·) is a feature concatenation operation, Concat(·) is used to and Taken together, EL(·) is a nonlinear activation function.

7. The method for automatically generating medical image diagnosis reports based on cross-modal global feature fusion Transformer according to claim 6 is characterized in that: Embedding visual features and text into X t Fusion, using the visual features V learned by the sliding window encoder in a multi-head cross-modal attention mechanism L As query, use the output of the masked attention mechanism As a key-value pair, it is expressed as follows: in, are three trainable parameter matrices, and MCA(·) stands for cross-modal attention mechanism; The global features are fused with the output of the cross-modal attention mechanism, and additional jump connections are introduced to protect the fragile information flow during the propagation process. The formula is expressed as: in, and g L are the outputs of FMCA and MRN, respectively.

8. The method for automatically generating medical image diagnosis reports based on cross-modal global feature fusion Transformer according to claim 7 is characterized in that: At the decoder layer L f Previously, global features were only fused with text features; then, f Starting from the layer, the global features are no longer learned together with the text features; In contrast, the FMCA module is responsible for extracting global features at this stage; the above operation formula is expressed as follows: Among them, l d Indicates the current layer number, l f Indicates the layer threshold for fusing text features, g L Represents the global information output by MRN, represents the text features after processing in the previous layer, and F' and F are the feature values ​​generated in the corresponding stages; Apply activation functions to feed-forward networks to generate corresponding reported results.

Images