Method for generating medical information data analysis report based on deep learning

Abstract

The application discloses a medical information data analysis report generation method based on deep learning, which comprises the following steps: acquiring medical image data and clinical text data and performing preprocessing to obtain preprocessed medical data; performing feature coding on the preprocessed medical data to obtain a multimodal feature set; generating a paired set based on the image region feature set and the text entity feature set, and calculating a pairing weight; grouping the paired set to obtain a paired subset set, and obtaining a centralized alignment vector pair set; solving an orthogonal rotation matrix by using Kabsch rigid optimal rotation alignment, and forming a rotation matrix set; calculating a difference degree and processing the rotation matrix set to obtain an updated rotation matrix set; and performing rotation alignment on the image region feature set to generate a medical information data analysis report. The application improves the accuracy of medical information data analysis report generation.

Description

Technical Field

[0001] This invention relates to the field of medical information processing technology, and in particular to a method for generating medical information data analysis reports based on deep learning. Background Technology

[0002] With the advancement of hospital informatization and imaging digitization, a large amount of medical imaging data and clinical text data are generated during the diagnosis and treatment process. How to jointly analyze imaging examination results, lesion information, and key medical record points to automatically generate structured and highly readable medical information data analysis reports has become a crucial requirement for smart healthcare. Existing technologies employ two approaches: one relies on extracting features separately through image processing and natural language processing, then generating reports through rule templates or simple concatenation; the other uses deep learning for end-to-end modeling of images or text, or joint representation learning within a multimodal framework, aiming to directly output report text or report elements. While these approaches reduce the cost of manual writing to some extent, they still face challenges in real clinical environments, including unstable multimodal information correspondences, insufficient interpretability, and limited generalization ability across examination conditions.

[0003] There is a natural cross-modal alignment requirement between lesion regions in medical images and medical entities in clinical texts. For example, abnormal image manifestations in the same examination site should be consistent with the corresponding entity descriptions. Existing multimodal report generation methods usually use attention mechanisms, similarity retrieval, or weakly supervised alignment to establish the association between image regions and text entities. However, this association is often implicit and probabilistic, and is easily affected by false detections of candidate regions, entity extraction noise, examination site confusion, and changes in sample distribution. This can lead to pairing drift and inconsistencies in cross-modal fusion, resulting in site mismatches, description discrepancies, or omissions of key information in the generated report. In addition, some methods do not explicitly quantify and weight the pairing quality, making it difficult to suppress the interference of low-confidence regions or low-confidence entities on the fusion process, resulting in large fluctuations in the generated results.

[0004] Therefore, how to provide a method for generating medical information data analysis reports based on deep learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a method for generating medical information data analysis reports based on deep learning. This invention uses deep learning to encode features of medical image data and clinical text data, obtains a stable pairing set through a cross-modal pairing mechanism with pairing weight constraints, and uses Kabsch rigid optimal rotation alignment to solve the orthogonal rotation matrix to achieve accurate alignment of image region features and text entity features. Combined with historical rotation matrix difference update rules, it suppresses drift and abnormal pairings, improves cross-modal fusion consistency and robustness, thereby improving the accuracy and reliability of medical information data analysis report generation.

[0006] The method for generating medical information data analysis reports based on deep learning according to embodiments of the present invention includes the following steps:

[0007] Acquire medical imaging data and clinical text data and perform preprocessing to obtain preprocessed medical data;

[0008] Feature encoding is performed on the preprocessed medical data to obtain a multimodal feature set;

[0009] A pairing set is generated based on the image region feature set and the text entity feature set, ensuring consistency of the examination site during generation, and the pairing weight is calculated.

[0010] The pairing set is grouped according to the inspection location to obtain a set of pairing subsets. An alignment vector pair set is generated based on each pairing subset, and a centering process is performed to obtain a set of centered alignment vector pairs.

[0011] For each paired subset, a weighted covariance relation is constructed for the set based on the pairing weight and using a centered alignment vector. The orthogonal rotation matrix that minimizes the weighted alignment error is solved by using Kabsch rigid optimal rotation alignment. A constraint with a positive determinant is applied to exclude mirror flipping, and the sets are collected to form a set of rotation matrices.

[0012] Obtain the historical set of rotation matrices and calculate the difference between the historical set of rotation matrices and the current set of rotation matrices. Then, process the set of rotation matrices according to the preset update rules to obtain the updated set of rotation matrices.

[0013] The image region feature set is rotated and aligned based on the updated set of rotation matrices, and a medical information data analysis report is generated based on the rotation alignment results.

[0014] Optionally, obtaining the preprocessed medical data specifically includes:

[0015] Acquire medical image data, establish raw image dataset, and sequentially perform de-identification, resampling, and intensity normalization processing on the raw image dataset to generate preprocessed image data;

[0016] Acquire clinical text data, establish a raw text dataset, perform de-identification and sentence / word segmentation on the raw text dataset to generate text sequences, perform medical entity extraction on the text sequences and generate entity sets;

[0017] Each medical entity in the entity set is standardized and encoded, and the standardized entity codes are then aggregated to form a standardized entity code set. The preprocessed image data, text sequence, and standardized entity code set are then combined to form preprocessed medical data.

[0018] Optionally, obtaining the multimodal feature set specifically includes:

[0019] Preprocessed image data, preprocessed text data, and standardized entity code sets are extracted from preprocessed medical data. The format and dimension consistency of the preprocessed image data and preprocessed text data are then checked to obtain coded image input and coded text input.

[0020] The coded image is fed into a deep learning image coding network to perform candidate region generation and feature extraction for each candidate region, outputting a set of image region features.

[0021] The coded text input and the standardized entity encoding set are fed into a deep learning text encoding network to perform segmented embedding and entity-aligned embedding operations, and semantic feature extraction is performed on each medical entity to output a text entity feature set.

[0022] The image region feature set and the text entity feature set are combined to form a multimodal feature set and then output.

[0023] Optionally, obtaining the pairing set and pairing weights specifically includes:

[0024] Read the region identifier, region feature vector, examination site label, and region detection confidence of the image region one by one from the image region feature set. Read the entity identifier, entity feature vector, examination site field, and entity extraction confidence of the medical entity one by one from the text entity feature set. Perform consistency verification and numerical validity verification on the region feature vector and entity feature vector to obtain the region data item and entity data item.

[0025] Using region data items and entity data items as input, perform a consistent screening operation for examination sites. Compare the examination site label of the image region with the examination site field of the medical entity and retain only the combinations that are consistent. Write the retained combinations into the candidate pairing set and record the corresponding region identifier and entity identifier. Calculate the similarity for each combination in the candidate pairing set.

[0026] Taking the candidate pair set and the similarity of each candidate pair as input, the region detection confidence and entity extraction confidence corresponding to the candidate pair are read. The similarity, region detection confidence and entity extraction confidence are multiplied by a preset non-negative weight coefficient and summed to obtain the pair weight. The consistency screening result, similarity calculation result and pair weight calculation result are combined and output as the pair set.

[0027] Optionally, obtaining the set of centered alignment vector pairs specifically includes:

[0028] Perform part grouping operation on the pairing set, create a container for each inspection part and traverse the pairing set, write the pairing items with the same inspection part into the same container to form a pairing subset set, generate a pairing subset identifier for each container and record the correspondence between the pairing subset identifier and the inspection part;

[0029] For any pairing subset, based on the region identifier and entity identifier of each pairing item within the pairing subset, the corresponding region feature vector and entity feature vector are retrieved from the multimodal feature set respectively, and then paired and combined one by one according to the order of the pairing items to generate an alignment vector pair set.

[0030] The alignment vector pair set is centered. All regional feature vectors in the alignment vector pair set are summarized and the regional mean vector is calculated. All entity feature vectors in the alignment vector pair set are summarized and the entity mean vector is calculated. At the same time, the regional mean removal operation is performed and the entity mean removal operation is performed on the entity feature vectors. All mean-removed vector pairs are summarized to obtain the centered alignment vector pair set.

[0031] Optionally, the formation of the set of rotation matrices specifically includes:

[0032] Read the set of centered alignment vector pairs and the pairing weights of any paired subset, extract the image-side centered vector and the corresponding text-side centered vector in the order of the vector pairs, and arrange the pairing weights in the same order to form a weight sequence;

[0033] For each vector pair, perform the outer product calculation to obtain the outer product matrix, and then weight the outer product matrix according to the corresponding pair weights to obtain the weighted outer product matrix. Finally, sum all the weighted outer product matrices in the order of the vector pairs to obtain the weighted covariance matrix.

[0034] Kabsch rigid optimal rotation alignment is used to perform singular value decomposition on the weighted covariance matrix to obtain the first orthogonal matrix, the singular value diagonal matrix, and the second orthogonal matrix. The candidate rotation matrix is ​​obtained by multiplying the second orthogonal matrix with the transpose of the first orthogonal matrix.

[0035] Calculate the determinant of the candidate rotation matrix and determine its sign. If the determinant is positive, the candidate rotation matrix is ​​determined as the rotation matrix. If the determinant is negative, the sign of the last column vector of the second orthogonal matrix is ​​flipped, and the second orthogonal matrix is ​​multiplied by the transpose of the first orthogonal matrix to obtain the rotation matrix with a positive determinant.

[0036] The rotation matrix is ​​associated with its corresponding paired subset and written into the rotation matrix set. The operations of reading vector pairs and weights, constructing a weighted covariance matrix, performing singular value decomposition and matrix multiplication, and performing determinant detection and sign flipping are repeated for all paired subsets to form the rotation matrix set.

[0037] Optionally, obtaining the updated set of rotation matrices specifically includes:

[0038] Obtain the set of historical rotation matrices associated with the target object, and perform consistency verification and matching operations on the set of historical rotation matrices and the set of rotation matrices. Based on the matching subset identifier, match the historical rotation matrices with the current rotation matrices one by one to obtain a list of matrix pairs.

[0039] For each matrix pair in the matrix pair list, perform sub-difference calculation, perform element-wise difference between the current rotation matrix and the historical rotation matrix to obtain the difference matrix, perform summation of squares and square root operation on each element of the difference matrix to obtain the sub-difference of the current matrix pair, and store the sub-difference as associated with the corresponding pair subset identifier;

[0040] The total difference is obtained by summing the sub-differences corresponding to all paired subset identifiers. In the case of missing historical rotation matrices or current rotation matrices, the missing items are added to the sum by a preset missing penalty value to obtain the difference used for update determination.

[0041] The rotation matrix set is processed according to a preset update rule, which includes a preset first threshold and a second threshold. When the difference is not greater than the first threshold, the rotation matrix set is not updated and the rotation matrix set is determined as the updated rotation matrix set. When the difference is greater than the first threshold but not greater than the second threshold, the pairing weights are updated, and the Kabsch rigid optimal rotation alignment solution is re-executed based on the updated pairing weights to obtain the updated rotation matrix set. When the difference is greater than the second threshold, the pairing set is updated, and a pairing subset set and a set of centered alignment vector pairs are regenerated based on the updated pairing set. The Kabsch rigid optimal rotation alignment solution is re-executed to obtain the updated rotation matrix set.

[0042] Optionally, the generation of the medical information data analysis report specifically includes:

[0043] Read the updated set of rotation matrices, build a rotation matrix index table based on the paired subset identifiers, traverse the set of image region features, select the corresponding rotation matrix in the rotation matrix index table based on the inspection site label, perform rotation transformation on the region feature vectors to obtain aligned region feature vectors, and collect them to form an aligned set of image region features.

[0044] Read the text entity feature set and perform cross-modal fusion processing to output a unified cross-modal representation;

[0045] Based on the preset report structure, paragraph planning is performed to obtain the examination findings paragraph structure and the conclusion paragraph structure. Based on the unified cross-modal representation, the examination findings paragraph structure is populated with content and the examination findings paragraph is output. Based on the unified cross-modal representation, the conclusion paragraph structure is populated with content and the conclusion paragraph is output. Based on the examination site label and paired subset identifier, the medical information data analysis report is merged and output.

[0046] The beneficial effects of this invention are:

[0047] After preprocessing and deep learning feature encoding of medical image data and clinical text data, this invention generates a pairing set of image regions and text entities with the consistency of the examination site as a constraint. It also calculates the pairing weight by combining similarity, region detection confidence and entity extraction confidence, thereby reducing the interference of false detection regions and noisy entities on cross-modal fusion from the source, making cross-modal association more reliable, and thus improving the accuracy of matching key information on which report generation depends.

[0048] This invention targets paired subsets obtained by grouping by inspection site, constructs a weighted covariance relationship based on pairing weights, and uses Kabsch rigid optimal rotation alignment to solve for an orthogonal rotation matrix that satisfies a positive determinant, thereby achieving deterministic alignment of image region features and text entity features in a unified coordinate space. This effectively reduces the drift and instability caused by implicit alignment and improves the consistency and reproducibility of cross-modal representation.

[0049] This invention introduces a historical rotation matrix set difference calculation and a preset update rule to adaptively process the rotation matrix set. This can suppress alignment offset and maintain alignment stability when continuously applied or when data distribution changes, thereby improving the accuracy, reliability and stability of medical information data analysis reports in terms of examination findings and conclusions. Attached Figure Description

[0050] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0051] Figure 1 This is a flowchart of the deep learning-based medical information data analysis report generation method proposed in this invention;

[0052] Figure 2 This diagram illustrates the Kabsch rigid optimal rotation alignment and rotation matrix update of the deep learning-based medical information data analysis report generation method proposed in this invention. Detailed Implementation

[0053] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0054] refer to Figures 1-2 A deep learning-based method for generating medical information data analysis reports includes the following steps:

[0055] Acquire medical imaging data and clinical text data and perform preprocessing to obtain preprocessed medical data;

[0056] Feature encoding is performed on the preprocessed medical data to obtain a multimodal feature set;

[0057] A pairing set is generated based on the image region feature set and the text entity feature set, ensuring consistency of the inspected parts during generation, and the pairing weight is calculated. The pairing weight is determined by similarity, region detection confidence, and entity extraction confidence.

[0058] The paired sets are grouped according to the inspection sites to obtain a set of paired subsets. An alignment vector pair set is generated based on each paired subset, and a centering process is performed to obtain a set of centered alignment vector pairs.

[0059] For each paired subset, a weighted covariance relation is constructed for the set based on the pairing weight and using a centered alignment vector. The orthogonal rotation matrix that minimizes the weighted alignment error is solved by using Kabsch rigid optimal rotation alignment. A constraint with a positive determinant is applied to exclude mirror flipping, and the sets are collected to form a set of rotation matrices.

[0060] Obtain the historical set of rotation matrices and calculate the difference between the historical set of rotation matrices and the current set of rotation matrices. Then, process the set of rotation matrices according to the preset update rules to obtain the updated set of rotation matrices.

[0061] The image region feature set is rotated and aligned based on the updated set of rotation matrices, and a medical information data analysis report is generated based on the rotation alignment results.

[0062] In this embodiment, obtaining the preprocessed medical data specifically includes:

[0063] Acquire medical image data, establish raw image dataset, and sequentially perform de-identification, resampling, and intensity normalization processing on the raw image dataset to generate preprocessed image data;

[0064] Acquire clinical text data, establish a raw text dataset, perform de-identification and sentence / word segmentation on the raw text dataset to generate text sequences, perform medical entity extraction on the text sequences and generate entity sets;

[0065] Standardized coding is performed on each medical entity in the entity set, and the standardized entity coding set is formed. The standardized coding generation process includes determining the examination site field and the lesion attribute field for each medical entity, and combining the preprocessed image data, text sequence, and standardized entity coding set to form preprocessed medical data.

[0066] In this embodiment, obtaining the multimodal feature set specifically includes:

[0067] Preprocessed image data, preprocessed text data, and standardized entity code sets are extracted from preprocessed medical data. The format and dimension consistency of the preprocessed image data and preprocessed text data are then checked to obtain coded image input and coded text input.

[0068] The coded image is input into a deep learning image coding network, which performs candidate region generation and feature extraction for each candidate region, and outputs a set of image region features. The deep learning image coding network is a coding network composed of an image input module, a region generation module, a region feature extraction module, a feature aggregation module, and a feature output module. The image input module performs tensor quantization and scale alignment operations, the region generation module performs candidate region generation or grid partitioning operations, the region feature extraction module performs convolutional feature extraction and region feature sampling operations, the feature aggregation module performs pooling and stitching operations, and the feature output module performs linear mapping operations and outputs a set of image region features.

[0069] Encodingable text input and a standardized entity encoding set are fed into a deep learning text encoding network to perform segmented embedding and entity alignment embedding operations. Semantic feature extraction is performed on each medical entity, and a text entity feature set is output. The deep learning text encoding network is an encoding network composed of a text input module, a word embedding module, an entity alignment module, a context encoding module, and a feature output module. The text input module performs serialization and truncation and padding operations, the word embedding module performs word vector mapping operations, the entity alignment module performs entity boundary localization and entity encoding fusion operations, the context encoding module performs sequence semantic modeling operations, and the feature output module performs linear mapping operations and outputs a text entity feature set.

[0070] The image region feature set and the text entity feature set are combined to form a multimodal feature set and then output.

[0071] In this embodiment, obtaining the pairing set and pairing weight specifically includes:

[0072] Read the region identifier, region feature vector, examination site label, and region detection confidence of the image region one by one from the image region feature set. Read the entity identifier, entity feature vector, examination site field, and entity extraction confidence of the medical entity one by one from the text entity feature set. Perform consistency verification and numerical validity verification on the region feature vector and entity feature vector to obtain the region data item and entity data item.

[0073] Using region data items and entity data items as input, a consistency screening operation for examination sites is performed. The examination site label of the image region is compared with the examination site field of the medical entity, and only combinations that are consistent are retained. The retained combinations are written into the candidate pairing set and the corresponding region identifier and entity identifier are recorded. For each combination in the candidate pairing set, a similarity is calculated. The similarity calculation operation includes performing a vector inner product calculation on the region feature vector and entity feature vector corresponding to the read combination to obtain the inner product value. The magnitude of the region feature vector and the magnitude of the entity feature vector are calculated respectively to obtain the first magnitude and the second magnitude. The similarity is obtained by dividing the inner product value by the product of the first magnitude and the second magnitude.

[0074] Using the candidate pairing set and the similarity of each candidate pairing as input, the region detection confidence and entity extraction confidence corresponding to the candidate pairing are read. The similarity, region detection confidence, and entity extraction confidence are multiplied by a preset non-negative weight coefficient and summed to obtain the pairing weight. The preset non-negative weight coefficient satisfies the summation value of one. The consistency screening result, similarity calculation result, and pairing weight calculation result are combined and output as a pairing set.

[0075] This invention introduces a consistent screening mechanism for examination sites and a pairing weighting mechanism based on similarity, region detection confidence, and entity extraction confidence between image region features and text entity features. This effectively filters out cross-site mismatches and low-confidence pairings and suppresses noise propagation, improving the accuracy and stability of the association between image regions and medical entities. This provides a more reliable pairing basis for cross-modal alignment and the generation of medical information data analysis reports.

[0076] In this embodiment, obtaining the set of centered alignment vector pairs specifically includes:

[0077] Perform part grouping operation on the pairing set, create a container for each inspection part and traverse the pairing set, write the pairing items with the same inspection part into the same container to form a pairing subset set, generate a pairing subset identifier for each container and record the correspondence between the pairing subset identifier and the inspection part;

[0078] For any pairing subset, based on the region identifier and entity identifier of each pairing item in the pairing subset, the corresponding region feature vector and entity feature vector are retrieved from the multimodal feature set respectively, and they are paired and combined one by one according to the order of the pairing items to generate an alignment vector pair set. Each vector pair in the alignment vector pair set is simultaneously associated with the pairing weight of the current pairing item and the order of the vector pairs is consistent with the order of the pairing items.

[0079] The alignment vector pair set is centered. All regional feature vectors in the alignment vector pair set are summarized and the regional mean vector is calculated. All entity feature vectors in the alignment vector pair set are summarized and the entity mean vector is calculated. At the same time, the regional mean removal operation is performed and the entity mean removal operation is performed on the entity feature vectors. All mean-removed vector pairs are summarized to obtain the centered alignment vector pair set.

[0080] In this embodiment, the formation of the rotation matrix set specifically includes:

[0081] Read the set of centered alignment vector pairs and the pairing weights of any paired subset, extract the image-side centered vector and the corresponding text-side centered vector in the order of the vector pairs, and arrange the pairing weights in the same order to form a weight sequence;

[0082] For each vector pair, perform the outer product calculation to obtain the outer product matrix, and then weight the outer product matrix according to the corresponding pair weights to obtain the weighted outer product matrix. Finally, sum all the weighted outer product matrices in the order of the vector pairs to obtain the weighted covariance matrix.

[0083] Kabsch rigid optimal rotation alignment is used to perform singular value decomposition on the weighted covariance matrix to obtain the first orthogonal matrix, the singular value diagonal matrix, and the second orthogonal matrix. The candidate rotation matrix is ​​obtained by multiplying the second orthogonal matrix with the transpose of the first orthogonal matrix.

[0084] Calculate the determinant of the candidate rotation matrix and determine its sign. If the determinant is positive, the candidate rotation matrix is ​​determined as the rotation matrix. If the determinant is negative, the sign of the last column vector of the second orthogonal matrix is ​​flipped, and the second orthogonal matrix is ​​multiplied by the transpose of the first orthogonal matrix to obtain the rotation matrix with a positive determinant.

[0085] The rotation matrix is ​​associated with its corresponding paired subset and written into the rotation matrix set. The operations of reading vector pairs and weights, constructing a weighted covariance matrix, performing singular value decomposition and matrix multiplication, and performing determinant detection and sign flipping are repeated for all paired subsets to form the rotation matrix set.

[0086] This invention constructs a weighted covariance by introducing pairing weight constraints within paired subsets, and uses Kabsch rigid optimal rotation alignment to solve for orthogonal rotation matrices that satisfy positive determinant, forming a set of rotation matrices corresponding to the grouping of examination sites. This achieves deterministic alignment of image-side features and text-side features in a unified coordinate space and eliminates errors caused by mirror flipping, improving the stability, consistency, and reproducibility of cross-modal alignment and enhancing the accuracy of fusion and medical information data analysis report generation.

[0087] In this embodiment, obtaining the updated set of rotation matrices specifically includes:

[0088] Obtain the set of historical rotation matrices associated with the target object, and perform consistency verification and matching operations on the set of historical rotation matrices and the set of rotation matrices. Based on the matching subset identifier, match the historical rotation matrices with the current rotation matrices one by one to obtain a list of matrix pairs.

[0089] For each matrix pair in the matrix pair list, perform sub-difference calculation, perform element-wise difference between the current rotation matrix and the historical rotation matrix to obtain the difference matrix, perform summation of squares and square root operation on each element of the difference matrix to obtain the sub-difference of the current matrix pair, and store the sub-difference as associated with the corresponding pair subset identifier;

[0090] The total difference is obtained by summing the sub-differences corresponding to all paired subset identifiers. In the case of missing historical rotation matrices or current rotation matrices, the missing items are added to the sum by a preset missing penalty value to obtain the difference used for update determination.

[0091] The rotation matrix set is processed according to a preset update rule, which includes a preset first threshold and a second threshold. When the difference is not greater than the first threshold, the rotation matrix set is not updated and the rotation matrix set is determined as the updated rotation matrix set. When the difference is greater than the first threshold but not greater than the second threshold, the pairing weights are updated, and the Kabsch rigid optimal rotation alignment solution is re-executed based on the updated pairing weights to obtain the updated rotation matrix set. When the difference is greater than the second threshold, the pairing set is updated, and a pairing subset set and a set of centered alignment vector pairs are regenerated based on the updated pairing set. The Kabsch rigid optimal rotation alignment solution is re-executed to obtain the updated rotation matrix set.

[0092] This invention introduces a difference measure between the historical rotation matrix set and the current rotation matrix set, combined with a missing value penalty mechanism, to achieve continuous consistency monitoring of cross-modal alignment status. It also selectively updates the pairing weights or pairing sets according to a preset threshold update rule to resolve the rotation matrix set, thereby suppressing alignment drift caused by abnormal samples and distribution changes and maintaining the stability and controllability of rotation alignment, thus improving the long-term accuracy and reliability of cross-modal fusion and medical information data analysis report generation.

[0093] In this embodiment, the generation of the medical information data analysis report specifically includes:

[0094] Read the updated set of rotation matrices, build a rotation matrix index table based on the paired subset identifiers, traverse the set of image region features, select the corresponding rotation matrix in the rotation matrix index table based on the inspection site label, perform rotation transformation on the region feature vectors to obtain aligned region feature vectors, and collect them to form an aligned set of image region features.

[0095] The text entity feature set is read and cross-modal fusion processing is performed. The cross-modal fusion processing includes serializing the aligned image region feature set according to the region identifier and performing feature projection to obtain image-side candidate representations, serializing the text entity feature set according to the entity identifier and performing feature projection to obtain text-side candidate representations, performing interactive alignment between the image-side candidate representations and the text-side candidate representations and generating interactive weights, performing normalization on the interactive weights and performing weighted aggregation, and outputting a unified cross-modal representation.

[0096] Based on the preset report structure, paragraph planning is performed to obtain the examination findings paragraph structure and the conclusion paragraph structure. Based on the unified cross-modal representation, the examination findings paragraph structure is filled with content and the examination findings paragraph is output. Based on the unified cross-modal representation, the conclusion paragraph structure is filled with content and the conclusion paragraph is output. Based on the examination site label and paired subset identifier, the medical information data analysis report is merged and output. The medical information data analysis report includes basic examination information, examination findings paragraph, lesion description paragraph, quantitative indicator paragraph, comparative analysis paragraph, risk warning paragraph, diagnostic conclusion paragraph, and treatment suggestion paragraph.

[0097] Example 1: To verify the feasibility of this invention in practice, it was applied to a real business scenario requiring the automatic generation of structured medical information data analysis reports. In this scenario, after a patient completes an imaging examination, medical imaging data is generated. Simultaneously, the clinical side generates clinical text data such as examination request information, chief complaint, present medical history, past medical history, laboratory summary, and preliminary diagnosis and treatment record. Traditionally, doctors rely on repeatedly switching between the imaging workstation and the medical record system and manually integrating key points, which easily leads to problems such as confusion of examination sites, inconsistencies between lesion descriptions and text entities, and omissions in comparative descriptions. Especially when there are multiple examination sites for the same patient, the unstable association between imaging regions and medical entities can lead to defects in the report content, such as mismatched sites, inconsistent descriptions, and unstable conclusions. The core problem that this invention aims to solve is the decrease in report accuracy and consistency caused by unstable cross-modal pairing and alignment, as well as the long-term unreliability caused by alignment drift during continuous operation.

[0098] In this scenario, the present invention interfaces with an image archiving system and a medical record text transfer system, automatically extracting medical image data and clinical text data for each examination and performing de-identification processing. On the image side, the original sequence is resampled and intensity normalized to obtain preprocessed image data. On the text side, the original record is segmented into sentences and words, and medical entities are extracted to form an entity set. Then, the entity set is standardized and encoded to generate a standardized entity code set. The examination site field and lesion attribute field are supplemented for each medical entity to provide subsequent consistency clues. Subsequently, the preprocessed image data is input into a deep learning image encoding network to output an image region feature set, and the preprocessed text and standardized entity code set are input into a deep learning text encoding network to output a text entity feature set. Finally, they are combined to form a multimodal feature set for subsequent pairing and alignment.

[0099] In the cross-modal pairing stage, this invention generates candidate pairings with the consistency of the inspection location as a constraint, and calculates the pairing weight by combining the similarity between the region feature vector and the entity feature vector, the region detection confidence and the entity extraction confidence, thereby reducing the disturbance of low-confidence candidates to subsequent alignment; in the alignment preparation stage, the pairing set is grouped into a pairing subset set according to the inspection location, and an alignment vector pair set is generated based on each pairing subset and a centering process is performed to eliminate translational deviations within the subset and highlight rotational relationships.

[0100] In the core alignment stage, this invention constructs a weighted covariance relationship based on pairing weights in each pairing subset. The weighted covariance matrix is ​​obtained by weighted external accumulation. Singular value decomposition is performed on the matrix, and candidate rotation matrices are calculated from two sets of orthogonal bases. Mirror flips are then eliminated by determinant detection and sign flipping to obtain orthogonal rotation matrices with positive determinants, which are then aggregated to form a rotation matrix set. To suppress alignment drift during continuous operation, a historical rotation matrix set is introduced and matched with the current rotation matrix set. Sub-difference is obtained by summing the squares and taking the square root of the element-wise differences, and then accumulated to form the overall difference. Missing historical items are penalized with a preset missing value, and the pairing weights or pairing sets are selectively updated according to a preset update rule. The rotation matrix set is then re-solved to maintain alignment stability.

[0101] During the report generation stage, this invention performs rotation alignment on the image region feature set based on the updated rotation matrix set, so that the image region features and text entity features are in a unified coordinate space. The aligned image-side representation and text-side representation are interactively aligned, weighted normalized, and weighted aggregated to obtain a unified cross-modal representation. This generates a medical information data analysis report that includes basic examination information, examination findings paragraphs, lesion description paragraphs, quantitative indicator paragraphs, comparative analysis paragraphs, risk warning paragraphs, diagnostic conclusion paragraphs, and treatment suggestion paragraphs. It also maintains a consistent mapping between the examination site and the paired subset to reduce site miswriting and entity mismatch.

[0102] To quantify and verify the beneficial effects, an evaluation set was constructed by selecting accumulated image examination samples and corresponding clinical text samples from the same business flow during continuous operation. Control schemes were set up, including a report assembly scheme based solely on rule templates, a scheme using common cross-modal attention alignment but without using Kabsch deterministic rotation alignment, and the scheme of this invention. Specific comparative data are shown in Table 1.

[0103] Partial consistency accuracy 86.3% 91.7% 96.8% Entity-Region Correspondence Accuracy 79.4% 87.2% 94.5% Accuracy of key lesion attributes 81.1% 88.6% 93.9% Comparative analysis of paragraph completeness rate 62.5% 78.9% 90.7% Effectiveness of risk warning paragraphs 58.2% 74.1% 88.3% Total time for generation and review (seconds / example) 520 360 215 Manual modification rate 24.6% 14.8% 6.9% Terminology inconsistency and cross-referencing rate 12.3% 6.4% 2.1%

[0104] As shown in Table 1, compared with the rule template assembly scheme and common cross-modal attention alignment schemes, the present invention performs better in terms of report quality indicators. The accuracy rate of part consistency increased from 86.3% and 91.7% to 96.8%, the accuracy rate of entity-region correspondence increased from 79.4% and 87.2% to 94.5%, and the accuracy rate of key lesion attributes increased from 81.1% and 88.6% to 93.9%. At the same time, the paragraph completeness rate and the effectiveness rate of risk warning paragraphs increased to 90.7% and 88.3%, respectively. In terms of efficiency indicators, the total time for generation and review was reduced to 215 seconds / case, and the manual modification rate was reduced to 6.9%. Moreover, the drift trigger rate was only 4.2%, which is lower than the 9.7% of common cross-modal attention alignment schemes. This shows that the present invention significantly improves the accuracy, completeness, and usability of medical information data analysis report generation while improving the stability and consistency of cross-modal alignment.

[0105] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for generating medical information data analysis reports based on deep learning, characterized in that, Includes the following steps: Acquire medical imaging data and clinical text data and perform preprocessing to obtain preprocessed medical data; Feature encoding is performed on the preprocessed medical data to obtain a multimodal feature set; A pairing set is generated based on the image region feature set and the text entity feature set, ensuring consistency of the examination site during generation, and the pairing weight is calculated. The paired sets are grouped according to the inspection sites to obtain a set of paired subsets. An alignment vector pair set is generated based on each paired subset, and a centering process is performed to obtain a set of centered alignment vector pairs. For each paired subset, a weighted covariance relation is constructed for the set based on the pairing weight and using a centered alignment vector. The orthogonal rotation matrix that minimizes the weighted alignment error is solved by using Kabsch rigid optimal rotation alignment. A constraint with a positive determinant is applied to exclude mirror flipping, and the sets are collected to form a set of rotation matrices. Obtain the historical set of rotation matrices and calculate the difference between the historical set of rotation matrices and the current set of rotation matrices. Then, process the set of rotation matrices according to the preset update rules to obtain the updated set of rotation matrices. The image region feature set is rotated and aligned based on the updated set of rotation matrices, and a medical information data analysis report is generated based on the rotation alignment results.

2. The method for generating medical information data analysis reports based on deep learning according to claim 1, characterized in that, The acquisition of the preprocessed medical data specifically includes: Acquire medical image data, establish raw image dataset, and sequentially perform de-identification, resampling, and intensity normalization processing on the raw image dataset to generate preprocessed image data; Acquire clinical text data, establish a raw text dataset, perform de-identification and sentence / word segmentation on the raw text dataset to generate text sequences, perform medical entity extraction on the text sequences and generate entity sets; Each medical entity in the entity set is standardized and encoded, and the standardized entity codes are then aggregated to form a standardized entity code set. The preprocessed image data, text sequence, and standardized entity code set are then combined to form preprocessed medical data.

3. The method for generating medical information data analysis reports based on deep learning according to claim 1, characterized in that, The acquisition of the multimodal feature set specifically includes: Preprocessed image data, preprocessed text data, and standardized entity code sets are extracted from preprocessed medical data. The format and dimension consistency of the preprocessed image data and preprocessed text data are then checked to obtain coded image input and coded text input. The coded image is fed into a deep learning image coding network to perform candidate region generation and feature extraction for each candidate region, outputting a set of image region features. The coded text input and the standardized entity encoding set are fed into a deep learning text encoding network to perform segmented embedding and entity-aligned embedding operations, and semantic feature extraction is performed on each medical entity to output a text entity feature set. The image region feature set and the text entity feature set are combined to form a multimodal feature set and then output.

4. The method for generating medical information data analysis reports based on deep learning according to claim 1, characterized in that, The specific steps for obtaining the pairing set and pairing weights include: Read the region identifier, region feature vector, examination site label, and region detection confidence of the image region one by one from the image region feature set. Read the entity identifier, entity feature vector, examination site field, and entity extraction confidence of the medical entity one by one from the text entity feature set. Perform consistency verification and numerical validity verification on the region feature vector and entity feature vector to obtain the region data item and entity data item. Using region data items and entity data items as input, perform a consistent screening operation for examination sites. Compare the examination site label of the image region with the examination site field of the medical entity and retain only the combinations that are consistent. Write the retained combinations into the candidate pairing set and record the corresponding region identifier and entity identifier. Calculate the similarity for each combination in the candidate pairing set. Taking the candidate pair set and the similarity of each candidate pair as input, the region detection confidence and entity extraction confidence corresponding to the candidate pair are read. The similarity, region detection confidence and entity extraction confidence are multiplied by a preset non-negative weight coefficient and summed to obtain the pair weight. The consistency screening result, similarity calculation result and pair weight calculation result are combined and output as the pair set.

5. The method for generating medical information data analysis reports based on deep learning according to claim 1, characterized in that, The specific steps to obtain the set of centered alignment vectors include: Perform part grouping operation on the pairing set, create a container for each inspection part and traverse the pairing set, write the pairing items with the same inspection part into the same container to form a pairing subset set, generate a pairing subset identifier for each container and record the correspondence between the pairing subset identifier and the inspection part; For any pairing subset, based on the region identifier and entity identifier of each pairing item within the pairing subset, the corresponding region feature vector and entity feature vector are retrieved from the multimodal feature set respectively, and then paired and combined one by one according to the order of the pairing items to generate an alignment vector pair set. The alignment vector pair set is centered. All regional feature vectors in the alignment vector pair set are summarized and the regional mean vector is calculated. All entity feature vectors in the alignment vector pair set are summarized and the entity mean vector is calculated. At the same time, the regional mean removal operation is performed and the entity mean removal operation is performed on the entity feature vectors. All mean-removed vector pairs are summarized to obtain the centered alignment vector pair set.

6. The method for generating medical information data analysis reports based on deep learning according to claim 1, characterized in that, The formation of the set of rotation matrices specifically includes: Read the set of centered alignment vector pairs and the pairing weights of any paired subset, extract the image-side centered vector and the corresponding text-side centered vector in the order of the vector pairs, and arrange the pairing weights in the same order to form a weight sequence; For each vector pair, perform the outer product calculation to obtain the outer product matrix, and then weight the outer product matrix according to the corresponding pair weights to obtain the weighted outer product matrix. Finally, sum all the weighted outer product matrices in the order of the vector pairs to obtain the weighted covariance matrix. Kabsch rigid optimal rotation alignment is used to perform singular value decomposition on the weighted covariance matrix to obtain the first orthogonal matrix, the singular value diagonal matrix, and the second orthogonal matrix. The candidate rotation matrix is ​​obtained by multiplying the second orthogonal matrix with the transpose of the first orthogonal matrix. Calculate the determinant of the candidate rotation matrix and determine its sign. If the determinant is positive, the candidate rotation matrix is ​​determined as the rotation matrix. If the determinant is negative, the sign of the last column vector of the second orthogonal matrix is ​​flipped, and the second orthogonal matrix is ​​multiplied by the transpose of the first orthogonal matrix to obtain the rotation matrix with a positive determinant. The rotation matrix is ​​associated with its corresponding paired subset and written into the rotation matrix set. The operations of reading vector pairs and weights, constructing a weighted covariance matrix, performing singular value decomposition and matrix multiplication, and performing determinant detection and sign flipping are repeated for all paired subsets to form the rotation matrix set.

7. The method for generating medical information data analysis reports based on deep learning according to claim 1, characterized in that, The updated set of rotation matrices is obtained specifically through: Obtain the set of historical rotation matrices associated with the target object, and perform consistency verification and matching operations on the set of historical rotation matrices and the set of rotation matrices. Based on the matching subset identifier, match the historical rotation matrices with the current rotation matrices one by one to obtain a list of matrix pairs. For each matrix pair in the matrix pair list, perform sub-difference calculation, perform element-wise difference between the current rotation matrix and the historical rotation matrix to obtain the difference matrix, perform summation of squares and square root operation on each element of the difference matrix to obtain the sub-difference of the current matrix pair, and store the sub-difference as associated with the corresponding pair subset identifier; The total difference is obtained by summing the sub-differences corresponding to all paired subset identifiers. In the case of missing historical rotation matrices or current rotation matrices, the missing items are added to the sum by a preset missing penalty value to obtain the difference used for update determination. The rotation matrix set is processed according to a preset update rule, which includes a preset first threshold and a second threshold. When the difference is not greater than the first threshold, the rotation matrix set is not updated and the rotation matrix set is determined as the updated rotation matrix set. When the difference is greater than the first threshold but not greater than the second threshold, the pairing weights are updated, and the Kabsch rigid optimal rotation alignment solution is re-executed based on the updated pairing weights to obtain the updated rotation matrix set. When the difference is greater than the second threshold, the pairing set is updated, and a pairing subset set and a set of centered alignment vector pairs are regenerated based on the updated pairing set. The Kabsch rigid optimal rotation alignment solution is re-executed to obtain the updated rotation matrix set.

8. The method for generating medical information data analysis reports based on deep learning according to claim 1, characterized in that, The generation of the medical information data analysis report specifically includes: Read the updated set of rotation matrices, build a rotation matrix index table based on the paired subset identifiers, traverse the set of image region features, select the corresponding rotation matrix in the rotation matrix index table based on the inspection site label, perform rotation transformation on the region feature vectors to obtain aligned region feature vectors, and collect them to form an aligned set of image region features. Read the text entity feature set and perform cross-modal fusion processing to output a unified cross-modal representation; Based on the preset report structure, paragraph planning is performed to obtain the examination findings paragraph structure and the conclusion paragraph structure. Based on the unified cross-modal representation, the examination findings paragraph structure is populated with content and the examination findings paragraph is output. Based on the unified cross-modal representation, the conclusion paragraph structure is populated with content and the conclusion paragraph is output. Based on the examination site label and paired subset identifier, the medical information data analysis report is merged and output.

Images