Cross-modal association retrieval method and system for hospital archives

Abstract

The invention discloses a cross-modal association retrieval method and system for hospital archives, relates to the technical field of medical information processing, and provides the following scheme: acquiring medical images of a target object in each period under the same time axis and associated diagnosis and treatment texts thereof, detecting the medical images in the same period to generate a focus set, and storing the focus set in a database; and calculating physical coordinates and quantitative indexes of the lesions in the unified coordinate system, marking phase identifiers for the lesions according to sequence information generated by the medical image metadata, and performing semantic analysis on the diagnosis and treatment text in the period. According to the method, the focus set is generated by detecting the same period of medical images, and the diagnosis and treatment text is synchronously analyzed to generate the anchor semantic tag and the period semantic set, so that the defect that the image quantitative characteristics and the text semantic description are lack of automatic association is overcome, automatic calibration of the image characteristics and the text description in the position, the phase and the quantitative dimension is realized, and the accuracy of calibration is improved. And manual cross validation is replaced.

Description

Technical Field

[0001] This invention relates to the field of medical information processing technology, and more specifically, this application relates to a method and system for cross-modal association retrieval of hospital records. Background Technology

[0002] In the process of building a smart healthcare system, intelligent recommendation systems based on electronic health records (EHRs) have become core tools for assisting clinical decision-making. Existing solutions achieve information retrieval through structured fields (such as diagnostic codes and examination types) and keyword matching: medical imaging archive systems (PACS) independently store image sequences and metadata such as CT and MRI, while hospital information systems (HIS) manage unstructured data such as medical reports and medical orders. During retrieval, images and text records from the same period are associated through patient ID and timestamp index, and semantic tags of the query input are extracted using predefined rules or natural language processing technology, ultimately returning a discrete period archive set that meets the filtering conditions.

[0003] Existing hospital record retrieval technologies have shortcomings in cross-modal and temporal understanding: The quantitative indicators of lesions generated from images lack an automated correlation mechanism with the semantic descriptions of diagnostic texts, forcing doctors to manually cross-validate, which is not only inefficient but also prone to overlooking contradictory information. Simultaneously, the dynamic evolution of lesions over time lacks structured modeling, forcing doctors to rely on experience to piece together discrete stage data, making it difficult to accurately track disease progression and increasing the risk of misdiagnosis. Furthermore, the retrieval mechanism relies too heavily on keyword matching and fails to integrate multi-dimensional verification, such as consistency verification between image features and semantic tags, or assessment of the overlap between the query time window and the actual lesion trajectory. Ultimately, this results often deviate from actual clinical needs. Therefore, a cross-modal correlation retrieval method and system for hospital records is proposed to address this problem. Summary of the Invention

[0004] To address the aforementioned technical problems, this technical solution provides a method and system for cross-modal association retrieval of hospital records, resolving the issues raised in the background section.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] Firstly, this application provides a method for cross-modal association retrieval of hospital records, the method comprising:

[0007] Obtain medical images of the target object at different times along the same timeline and their associated diagnostic and treatment texts;

[0008] The same medical images are detected to generate a lesion set, the physical coordinates and quantitative indicators of each lesion are calculated in a unified coordinate system, and the phase identifier of each lesion is marked according to the sequence information generated by the medical image metadata.

[0009] Semantic parsing is performed on the diagnostic and treatment texts in this period to generate anchor semantic tags that describe the location, phase, or quantitative features of lesions, as well as a period semantic set that reflects the changes of lesions across periods;

[0010] Based on the lesion set and anchor point semantic tag set of the same period, the matching degree between the physical coordinates, phase identifiers and quantitative indicators of each lesion and the corresponding semantic tags is calculated and aggregated to form a lesion consistency score.

[0011] Coordinate system registration is performed on medical images of each stage. Based on physical coordinates and quantitative indicators, the trajectory of the same lesion evolving on the time axis is constructed. The semantic set of the associated stages is extracted and structured to obtain the semantic features of the stages.

[0012] The system obtains the query input and parses it into a set of query semantic tags and optional query period constraints. Based on the set of query semantic tags, it determines the trajectory of candidate lesions according to a preset filtering strategy.

[0013] For each candidate lesion trajectory, the trajectory consistency score is obtained by aggregating the lesion consistency scores of each stage. When there is a query time constraint, the correlation evaluation of the query stage constraint is performed based on the stage semantic features of the candidate lesion trajectory to generate a stage consistency score.

[0014] The trajectory consistency score and the period consistency score are aggregated according to the preset fusion model to form a retrieval score, and the candidate lesion trajectories are ranked accordingly.

[0015] Secondly, this application provides a cross-modal association retrieval system for hospital records, used to implement the cross-modal association retrieval method for hospital records described in any of the above claims, including:

[0016] The multi-source spatiotemporal alignment module is used to acquire medical images of the target object at different times along the same time axis and their associated diagnostic and treatment texts.

[0017] The lesion quantification and annotation module is used to detect and generate a lesion set from medical images of the same period, calculate the physical coordinates and quantification indicators of each lesion in a unified coordinate system, and annotate the phase identifier of each lesion based on the sequence information generated from the medical image metadata.

[0018] The text semantic parsing module is used to perform semantic parsing on the diagnosis and treatment text in this period, and generate anchor semantic tags that describe the location, phase or quantitative features of the lesion, as well as a period semantic set that reflects the changes of the lesion across periods.

[0019] The cross-modal consistency module is used to calculate the matching degree between the physical coordinates, phase identifiers and quantitative indicators of each lesion and the corresponding semantic tags based on the lesion set and anchor point semantic tag set of the same period, and aggregate them to form a lesion consistency score.

[0020] The lesion trajectory modeling module is used to perform coordinate system registration on medical images of different stages, construct the lesion trajectory of the same lesion on the time axis based on physical coordinates and quantitative indicators, and extract and structure the semantic set of the associated stages to obtain the semantic features of the stages.

[0021] The query parsing module is used to obtain query input and parse it into a set of query semantic tags and optional query period constraints, and determine the candidate lesion trajectory according to the preset filtering strategy based on the set of query semantic tags.

[0022] The trajectory scoring module is used to obtain a trajectory consistency score for each candidate lesion trajectory by aggregating the lesion consistency scores of each stage of the lesion. When there is a query time constraint, the module evaluates the relevance of the query stage constraint based on the stage semantic features of the candidate lesion trajectory and generates a stage consistency score.

[0023] The retrieval and ranking module is used to aggregate trajectory consistency scores and period consistency scores according to a preset fusion model to form a retrieval score, and to rank candidate lesion trajectories accordingly.

[0024] Thirdly, this application provides a computer device including a memory and a processor, the memory storing code, and the processor being configured to acquire the code and execute the aforementioned cross-modal association retrieval method for hospital records.

[0025] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned cross-modal association retrieval method for hospital records.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] This application generates a lesion set by detecting medical images of the same period and simultaneously analyzes the diagnosis and treatment text to generate anchor semantic tags and period semantic sets. This overcomes the lack of automatic association between image quantitative features and text semantic descriptions and realizes automatic calibration of image features and text descriptions in terms of position, phase and quantification dimensions, replacing manual cross-validation.

[0028] This application constructs lesion trajectories across stages through coordinate system registration and extracts structured stage semantic features by combining stage semantic sets, which solves the problem of lack of structured expression of dynamic evolution of lesions and realizes spatiotemporal integrated modeling of lesion progression.

[0029] This application solves the problem of relying on single keyword matching in the retrieval process by using a dual verification mechanism of lesion consistency score and stage consistency score, realizes multi-dimensional credibility verification of candidate lesion trajectory, and improves the matching rate between retrieval results and real clinical needs. Attached Figure Description

[0030] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Wherein:

[0031] Figure 1 The flowchart shows the cross-modal association retrieval method for hospital records proposed in this invention.

[0032] Figure 2 This is a structural block diagram of the cross-modal association retrieval system for hospital records proposed in this invention. Detailed Implementation

[0033] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0034] Reference Figure 1 As shown, this application proposes a cross-modal association retrieval method for hospital records, including:

[0035] Obtain medical images of the target object at different times along the same timeline and their associated diagnostic and treatment texts;

[0036] The system detects and generates a set of lesions from medical images of the same period, calculates the physical coordinates and quantitative indicators of each lesion in a unified coordinate system, and marks the phase of each lesion with sequence information generated from medical image metadata. The quantitative indicators refer to the objective numerical description of the lesions, including the volume, diameter, density (such as CT value), signal intensity (such as MRI), etc.

[0037] Semantic parsing is performed on the diagnostic and treatment texts in this period to generate anchor semantic tags that describe the location, phase, or quantitative features of lesions, as well as a period semantic set that reflects the changes of lesions across periods;

[0038] Based on the lesion set and anchor point semantic tag set of the same period, the matching degree between the physical coordinates, phase identifiers and quantitative indicators of each lesion and the corresponding semantic tags is calculated and aggregated to form a lesion consistency score.

[0039] Coordinate system registration was performed on medical images of each stage. Based on physical coordinates and quantitative indicators, the trajectory of the same lesion evolving on the time axis was constructed. The semantic set of the associated stages was extracted and structured to obtain the semantic features of the stages. Among them, coordinate system registration was performed by aligning images at different time points to the same anatomical space using image registration software.

[0040] The system obtains the query input and parses it into a set of query semantic tags and optional query period constraints. Based on the set of query semantic tags, it determines the trajectory of candidate lesions according to a preset filtering strategy.

[0041] For example, the preset filtering strategy may be:

[0042] Spatial preliminary screening: Only retain trajectories in which the physical coordinates of at least one stage of lesion fall within the anatomical region corresponding to the query semantic tag location element (error tolerance ≤ 10mm) to ensure spatial relevance;

[0043] Phase compatibility filtering: If querying a single phase (e.g., "arterial phase"), the phase identifier of lesions in ≥50% of phases in the trajectory must match the phase or a compatible phase (e.g., CT arterial phase compatible with portal venous phase). If querying multiple phases (e.g., "T1 or T2"), at least one phase in the trajectory must match any of the specified phases.

[0044] Dynamic scoring and ranking: Based on the distribution of query semantic tag types, weights are dynamically allocated (phase weight = number of phase tags / total number of tags, position weight = number of position tags / total number of tags, quantization weight = 1 - phase weight - position weight), and the approximation of each lesion trajectory with the tag set is calculated;

[0045] Threshold filtering: Eliminate trajectories with an approximation of <0.6, and retain the top K trajectories in descending order of approximation (K=min(50, total number of trajectories×20%)) as candidate lesion trajectories;

[0046] For each candidate lesion trajectory, the trajectory consistency score is obtained by aggregating the lesion consistency scores of each stage. When there is a query time constraint, the correlation evaluation of the query stage constraint is performed based on the stage semantic features of the candidate lesion trajectory to generate a stage consistency score.

[0047] The trajectory consistency score and the period consistency score are aggregated according to the preset fusion model to form a retrieval score, and the candidate lesion trajectories are ranked accordingly.

[0048] The preset fusion model is:

[0049] ;

[0050] In the formula, To obtain a score, To harmonize the weighting coefficients, Scoring is given based on trajectory consistency. The period consistency score; for example, when the candidate lesion trajectory contains lesions from the three most recent periods. A value of 0.6 can be taken; when the rate of change of the quantitative indicators of lesions is less than 5% for three consecutive periods, 0.3 is acceptable;

[0051] Specifically, this application addresses the limitations of existing medical record retrieval technologies in cross-modal and temporal understanding. By establishing an automated association between quantitative indicators of medical images and semantic descriptions of diagnostic and treatment texts, it achieves structured modeling of the temporal evolution of lesions. Through the integration of multi-dimensional verification, including consistency verification of image features and semantic tags, and assessment of the overlap between the query time window and the actual lesion trajectory, it can provide retrieval results that better meet real clinical needs.

[0052] In an optional embodiment, the generation of the stage semantic set reflecting the changes of lesions across stages specifically includes:

[0053] Identify relative time descriptions and lesion change descriptions in the current stage of diagnosis and treatment text;

[0054] It should be noted that medical records often contain doctors' descriptions of changes in lesions, and may also involve time information, i.e., relative time descriptions; such as "two weeks ago" and "at the last examination", as well as changes in the lesions, such as "grew larger than before", "new lesions", "lesions disappeared"; named entity recognition models can be used to identify time entities and verbs or phrases related to lesions in the text; for example, a pre-trained medical domain NLP model can identify "two weeks ago" as a relative time description, and "grew larger", "shrank", "newly appeared", "disappeared" as descriptions of changes in lesions;

[0055] Based on the timestamp of the current period, the relative time description is mapped to the target period number on the time axis;

[0056] When the description of lesion change includes lesion generation or disappearance events, the previous period closest to the current period on the timeline is used as the target period number.

[0057] Extract the location, direction of change, and magnitude of lesion changes from the description of lesion changes, and encode these three factors together into quantitative semantic features;

[0058] Construct structured triples using the current period number, the target period number, and quantitative semantic features, and aggregate all structured triples to form a period semantic set;

[0059] To facilitate understanding of the above embodiments, a specific application scenario of the above embodiments will be used as an example for illustration below:

[0060] Using a specific patient as the target, obtain their medical images and associated diagnostic and treatment texts for four different time periods on the same timeline:

[0061] Episode: T1, Timestamp: 2025-01-10, Medical Imaging: CT, Lesion Description: A 2.0cm low-density shadow was found in liver segment S8;

[0062] Stage: T2, Timestamp: 2025-02-15, Medical Imaging: MRI, Lesion Change Description: The lesion in liver segment S8 increased to 3.2cm;

[0063] Episode: T3, Timestamp: 2025-03-20, Medical Imaging: CT, Lesion Change Description: Newly developed 0.5cm cyst in liver S4 segment;

[0064] Stage: T4, Timestamp: 2025-06-01, Medical Imaging: MRI, Lesion Change Description: The lesion in liver segment S8 shrank to 2.8cm;

[0065] Using T3 as the current stage, the excerpt of the diagnosis and treatment text is as follows: Compared with the MRI image on February 15, 2025, the lesion in liver segment S8 increased by approximately 1.2 cm compared to the previous image; a new small cyst with a diameter of 0.5 cm was found in liver segment S4; the original metastatic lesion in segment S5 disappeared.

[0066] Identifying medical records yields relative time descriptions and lesion change events, including:

[0067] Relative time description: "compared to the previous time", which is mapped to the target period number T2;

[0068] Lesion alteration events:

[0069] The lesion in segment S8 of the liver was "enlarged" and "1.2cm" in size.

[0070] A cyst in liver segment S4, described as "new" and "0.5cm";

[0071] The metastatic lesion in liver segment S5 has "disappeared";

[0072] It should be noted that although all three lesions are mapped to T2, the cyst in liver segment S4 and the metastatic lesion in liver segment S5 are events of lesion formation or disappearance, so they are mapped to the previous T2 most recent stage of the current stage, while the lesion in liver segment S8 is mapped to T2 based on relative time description.

[0073] Finally, three structured triples are constructed:

[0074] {"Current Stage": "T3", "Target Stage": "T2", "Quantitative Semantic Features": {"Law Location": "Hepatic S8 Segment", "Direction of Change": "Increase", "Amplitude": "1.2cm"}};

[0075] {"Current Stage": "T3", "Target Stage": "T2", "Quantitative Semantic Features": {"Law Location": "Hepatic S4 Segment", "Direction of Change": "New", "Amplitude": "0.5cm"}};

[0076] {"Current Stage": "T3", "Target Stage": "T2", "Quantitative Semantic Features": {"Law Location": "Liver S5 Segment", "Direction of Change": "Disappeared", "Magnitude": "N / A"}};

[0077] It should be noted that by analyzing the relative time descriptions and lesion change events in the diagnostic and treatment texts, a structured semantic association across different periods is automatically constructed, solving the temporal misalignment problem that traditional methods rely on manual annotation. For example, it transforms the vague "increased compared to the previous time" into a precise...<T3→T2,+1.2cm> Quantifying semantic features provides computable temporal anchors for lesion trajectory construction;

[0078] Through the above technical solution, this embodiment provides a method for extracting and structuring cross-stage change information of lesions from medical texts, which provides a reliable semantic basis for subsequently constructing more accurate lesion trajectories and querying stage constraints, and improves the accuracy and intelligence level of cross-modal association retrieval.

[0079] In an optional embodiment, phase identifiers are marked for each lesion based on sequence information generated from parsed metadata, specifically including:

[0080] The medical images include CT images, MRI images, and ultrasound images;

[0081] For CT images, extract the protocol name and scan time from the metadata, determine the phase according to the first preset rule, and mark the phase as arterial phase, portal venous phase, or delayed phase.

[0082] For example, the first preset rules include: arterial phase, the criteria for determination is that the protocol name contains "ART" or the scan time is 25-40 seconds after contrast agent injection; portal venous phase, the criteria for determination is that the protocol name contains "PV" or the scan time is 60-80 seconds after contrast agent injection; and delayed phase, the criteria for determination is that the protocol name contains "Delay" or the scan time is >120 seconds.

[0083] For MRI images, the sequence name, ImageType and scan time are extracted from the metadata, and the second preset rule is used to determine and mark the phase as T1, T2, DWI or dynamic enhancement period;

[0084] For example, the second preset rule includes: T1, the determination condition is that the sequence name contains "T1" or the ImageType contains "T1W"; T2, the determination condition is that the sequence name contains "T2" or the ImageType contains "T2W"; DWI, the determination condition is that the sequence name contains "DWI" or "DIFFUSION"; dynamic enhancement phase, the determination condition is that the ImageType contains "DYNAMIC" and the scan time meets the arterial phase / portal venous phase standard;

[0085] For ultrasound images or medical images that fail to be determined, the phase identifier is marked as none;

[0086] Through the above technical solution, this embodiment can specifically analyze the metadata of different medical imaging modalities (such as CT images, MRI images and ultrasound images), and accurately determine their physiological or enhancement phase according to preset rules, ensuring the accuracy and standardization of lesion phase identification, improving the accuracy of cross-modal association retrieval, and enabling the system to more accurately locate lesions in different phases.

[0087] In an optional embodiment, the matching degree between the lesion and each semantic tag is calculated using physical coordinates, phase identifiers, and quantification indicators to generate a lesion consistency score, specifically including:

[0088] Extract the physical coordinates, phase identifiers and quantitative indicators of each lesion in the lesion set, and extract the semantic elements of each semantic tag in the anchor point semantic tag set, and record them as location elements, phase elements and quantitative elements.

[0089] Calculate the anatomical distance between physical coordinates and location features, map it to a score using a preset attenuation function, and use it as the location matching degree;

[0090] For example, the preset decay function is mapped as follows: ;in, For function mapping scores, Anatomical distance;

[0091] By comparing the phase identifier with the phase element, the three-state scores of equal, compatible or conflicting are output as the phase matching degree.

[0092] For example, based on a medical knowledge graph, the shortest path length between two phase nodes is queried. If the length = 0, they are considered equal; if the length = 1, they are considered compatible; if the length ≥ 2, they are considered conflicting. The scores for equality, compatibility, and conflict are 1.0, 0.5, and 0, respectively.

[0093] The quantitative indicators and quantitative elements are input into the Huber difference function to generate a score, which is then used as the quantitative matching degree.

[0094] For example, the Huber difference function is:

[0095] ;

[0096] In the formula, This is the Huber difference value. For quantitative indicators, To quantify the elements, The threshold parameter is set to 0.5cm;

[0097] The formula for calculating the quantitative matching degree is:

[0098] ;

[0099] in, To quantify the degree of matching, The maximum permissible difference is set at 2.0 cm;

[0100] The phase matching degree, location matching degree, and quantitative matching degree are weighted and aggregated. If the phase identifier is none, the phase matching degree weight is reset to zero and normalized according to the remaining weights, and the lesion consistency score is output. For example, the weight ratio of phase matching degree, location matching degree, and quantitative matching degree can be 3:2:5.

[0101] To facilitate understanding of the above embodiments, the application scenario of the above embodiments will be used as an example for illustration:

[0102] Using T3 as the current stage, it includes a set of lesions and a set of anchor semantic tags:

[0103] The lesions are clustered as follows:

[0104] Lesion 1: Location is in liver S8 segment (physical coordinates [120,45,80]), quantitative indicator is diameter 3.5cm, phase label is "arterial phase";

[0105] Lesion 2: Location is liver S4 segment (physical coordinates [90,30,60]), quantitative indicator is diameter 0.5cm, phase identifier is "arterial phase";

[0106] The anchor semantic tag set is as follows:

[0107] Semantic label 1: Location is liver S8 segment (mapped coordinates [118,46,82]), phase is "arterial phase", quantified as diameter 3.5cm;

[0108] Semantic label 2: Location is liver S4 segment (mapped coordinates [92,31,58]), phase is "arterial phase", quantified as diameter 0.5cm;

[0109] Taking lesion 1 and semantic label 1 as an example, perform a matching degree calculation:

[0110] Location matching degree calculation:

[0111] Anatomical distance ;

[0112] Preset decay function mapping: ;

[0113] Phase matching degree calculation:

[0114] Since both the phase identifier and the phase element are in the "arterial phase", the output is equal to the corresponding score, which is 1.0.

[0115] Quantitative matching degree calculation:

[0116] Input the quantitative indicator 3.5cm and the quantitative element 3.5cm into the Huber difference function, and the output is 1.0;

[0117] Weighted aggregation: 0.74×0.3+1.0×0.2+1.0×0.5=0.922 as the lesion consistency score;

[0118] It should be noted that robust calibration of cross-modal features is achieved by mapping the attenuation function of anatomical distance, determining the three phase states (equal / compatible / conflicting), and fusing the Huber difference function. When the phase label is absent, such as in ultrasound images, the weights are dynamically adjusted to avoid misjudgment and improve the accuracy of consistency assessment between lesions and semantic labels. Semantic labels are usually keywords in the query input.

[0119] Through the above technical solution, this embodiment can perform multi-dimensional quantitative evaluation of the matching degree between the physical coordinates, phase identifiers and quantitative indicators of lesions and corresponding semantic tags, and can more accurately identify the association between lesions in medical images and descriptions in diagnostic texts, thereby improving the accuracy of cross-modal association retrieval.

[0120] In an optional embodiment, the parsing of the query period constraint specifically includes:

[0121] Identify absolute period identifiers or relative time descriptions in the query input;

[0122] For absolute period identifiers, they are parsed into start period number and end period number to form absolute period pairs;

[0123] For relative time descriptions, they are mapped to the starting period number on the timeline relative to the current timestamp, and all possible period pairs from the starting period number to the current period number are constructed based on the preset examination frequency, forming a set of relative period pairs; where the preset examination frequency is the examination interval time input by the doctor based on experience, such as 14 days, 30 days, 90 days, etc.

[0124] The union of absolute period pairs and relative period pairs is output as the query period constraint;

[0125] To facilitate understanding of the above embodiments, the application scenario of the above embodiments will be used as an example for illustration:

[0126] The query input is: Retrieve enlarged liver lesions within the past six months. This is parsed into a set of semantic tags and optional query period constraints.

[0127] The semantic tags for the query are: "liver lesions" and "enlargement";

[0128] The process of obtaining the query period constraint is as follows:

[0129] The query input is identified as a relative time description: within the past three months;

[0130] Using the current timestamp 2025-06-01, the calculated start timestamp is 2024-12-01;

[0131] With a preset inspection frequency of 30 days, all possible period pairs include: [T1→T2], [T2→T3], [T3→T4], forming a relative period pair set. At this time, the absolute period pairs are empty, and the union of these pairs is the relative period pair set.

[0132] It should be noted that by dynamically mapping time windows and automatically generating consecutive period pairs, open time descriptions, such as "the past six months", are transformed into a computable discrete period sequence, breaking through the dependence of traditional retrieval systems on absolute timestamps and improving the adaptability of clinical fuzzy queries.

[0133] Through the above technical solution, this embodiment improves the flexibility and accuracy of queries by identifying and parsing different query period constraints in user query input and converting them into structured period pairs that the system can process. This allows users to specify the time range for retrieval in a more natural and clinically consistent manner according to their actual needs. By integrating different forms of period constraints into a unified query period constraint output, the subsequent screening and evaluation process is simplified, and the system is ensured to more accurately match user intent, enabling refined retrieval of lesion evolution information within a specific time period in medical records.

[0134] In one optional embodiment, the relevance of the query period constraint is evaluated based on the period semantic features of the candidate lesion trajectory to generate a period consistency score, specifically including:

[0135] Traverse each structured triplet in the semantic features of the issue, extract the current issue number and the target issue number to construct issue pairs, and form the actual issue set;

[0136] The overlap between the actual period set and the query period constraint is calculated and recorded as the period consistency score. The overlap is obtained by dividing the number of elements in the intersection by the number of elements in the union.

[0137] It should be noted that by traversing the structured triples in the semantic features of the period, an actual period set reflecting the actual evolution of the lesion is constructed, and the overlap between this actual period set and the query period constraints is calculated to generate a period consistency score. This quantifies the degree of matching between the lesion trajectory and the user's query intent in the time dimension, effectively solving the problem of how to accurately assess the temporal correlation of the trajectory.

[0138] To facilitate understanding of the above embodiments, the application scenario of the above embodiments will be used as an example for illustration:

[0139] The candidate lesion trajectory is: the trajectory of the lesion in liver segment S8. Its stage semantic features are extracted from the stage semantic sets in stages T3 and T4, and structured as follows:

[0140] [{"Current Stage":"T3","Target Stage":"T2","Quantitative Semantic Features":{"Law Location":"Liver S8 segment","Direction of Change":"Increase","Amplitude":"1.2cm"}},{"Current Stage":"T4","Target Stage":"T3","Quantitative Semantic Features":{"Law Location":"Liver S8 segment","Direction of Change":"Shrink","Amplitude":"0.4cm"}}];

[0141] Therefore, we can obtain the following: the actual period set is: {[T2,T3],[T3,T4]}; the query period constraint is: {[T1,T2],[T2,T3],[T3,T4]};

[0142] Calculate the period consistency score:

[0143] Intersection: {[T2,T3],[T3,T4]} → Number of elements = 2;

[0144] Union: {[T1,T2],[T2,T3],[T3,T4]} → Number of elements = 3;

[0145] The consistency score for each period is approximately 0.67 (2 / 3).

[0146] Through the above technical solution, after obtaining the query input and parsing it into a set of query semantic tags and optional query period constraints, when query period constraints exist, the relevance of the query period constraints can be accurately evaluated based on the period semantic features of the candidate lesion trajectory. This allows for more accurate screening and sorting of candidate lesion trajectories, ensuring that the search results not only match in semantic content but also highly meet the user's specific query needs in terms of time evolution features, thereby improving the accuracy of the search.

[0147] In an optional embodiment, after sorting the candidate lesion trajectories, a visual retrieval display is also included:

[0148] Visualize and output the medical images and diagnostic text fragments corresponding to the lesion trajectory in chronological order;

[0149] The visualization output includes trajectory consistency scores and retrieval scores, and the evolution features of lesion trajectories and the semantic features of stages are displayed side-by-side for comparison.

[0150] Specifically, medical images are displayed as sequential frames or key phases juxtaposed, overlaid with lesion annotations, physical coordinates, and quantitative indicators; diagnostic and treatment texts extract key paragraphs describing lesion size, location, and changes, which are displayed synchronously with the medical images and highlighted with semantic tags; the interface synchronously displays trajectory consistency scores and retrieval relevance scores, with explanations of the score composition to enhance interpretability; simultaneously, the image evolution features of lesion trajectories (overlay comparison of lesion regions in multiple phases) and phase semantic features (structured triples, such as phase number-quantitative features-change descriptions) are juxtaposed, realizing dual-channel mutual verification of morphological changes in medical images and semantic changes in diagnostic and treatment texts, providing a spatiotemporal consistency cognitive framework for lesion evolution;

[0151] Through the above technical solution, after sorting the candidate lesion trajectories, the search results can be displayed in an intuitive and comprehensive way. Not only can the lesion trajectories ranked first be seen, but the evolution characteristics of the lesions in different stages of medical images, such as morphology, size, and location, can also be clearly understood through visualization output, as well as the semantic description of these changes in the diagnostic and treatment text. This improves the interpretability of the search results, reduces the cognitive burden of understanding the evolution process of complex lesions, and improves the efficiency and accuracy of medical diagnosis and decision-making.

[0152] See Figure 2 As shown, this solution proposes a cross-modal association retrieval system for hospital records, used to implement the aforementioned cross-modal association retrieval method for hospital records, including:

[0153] The multi-source spatiotemporal alignment module is used to acquire medical images of the target object at different times along the same time axis and their associated diagnostic and treatment texts.

[0154] The lesion quantification and annotation module is used to detect and generate a lesion set from medical images of the same period, calculate the physical coordinates and quantification indicators of each lesion in a unified coordinate system, and annotate the phase identifier of each lesion based on the sequence information generated from the medical image metadata.

[0155] The text semantic parsing module is used to perform semantic parsing on the diagnosis and treatment text in this period, and generate anchor semantic tags that describe the location, phase or quantitative features of the lesion, as well as a period semantic set that reflects the changes of the lesion across periods.

[0156] The cross-modal consistency module is used to calculate the matching degree between the physical coordinates, phase identifiers and quantitative indicators of each lesion and the corresponding semantic tags based on the lesion set and anchor point semantic tag set of the same period, and aggregate them to form a lesion consistency score.

[0157] The lesion trajectory modeling module is used to perform coordinate system registration on medical images of different stages, construct the lesion trajectory of the same lesion on the time axis based on physical coordinates and quantitative indicators, and extract and structure the semantic set of the associated stages to obtain the semantic features of the stages.

[0158] The query parsing module is used to obtain query input and parse it into a set of query semantic tags and optional query period constraints, and determine the candidate lesion trajectory according to the preset filtering strategy based on the set of query semantic tags.

[0159] The trajectory scoring module is used to obtain a trajectory consistency score for each candidate lesion trajectory by aggregating the lesion consistency scores of each stage of the lesion. When there is a query time constraint, the module evaluates the relevance of the query stage constraint based on the stage semantic features of the candidate lesion trajectory and generates a stage consistency score.

[0160] The retrieval and ranking module is used to aggregate trajectory consistency scores and period consistency scores according to a preset fusion model to form a retrieval score, and to rank candidate lesion trajectories accordingly.

[0161] In another embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above embodiments.

[0162] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps described above.

[0163] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps described above.

[0164] The technical scope of this invention is not limited to the content described above. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the protection scope of this invention.

Claims

1. A cross-modal association retrieval method for hospital records, characterized in that, The method includes: Obtain medical images of the target object at different times along the same timeline and their associated diagnostic and treatment texts; The same medical images are detected to generate a lesion set, the physical coordinates and quantitative indicators of each lesion are calculated in a unified coordinate system, and the phase identifier of each lesion is marked according to the sequence information generated by the medical image metadata. Semantic parsing is performed on the diagnostic and treatment texts in this period to generate anchor semantic tags that describe the location, phase, or quantitative features of lesions, as well as a period semantic set that reflects the changes of lesions across periods; Based on the lesion set and anchor point semantic tag set of the same period, the matching degree between the physical coordinates, phase identifiers and quantitative indicators of each lesion and the corresponding semantic tags is calculated and aggregated to form a lesion consistency score. Coordinate system registration is performed on medical images of each stage. Based on physical coordinates and quantitative indicators, the trajectory of the same lesion evolving on the time axis is constructed. The semantic set of the associated stages is extracted and structured to obtain the semantic features of the stages. The system obtains the query input and parses it into a set of query semantic tags and optional query period constraints. Based on the set of query semantic tags, it determines the trajectory of candidate lesions according to a preset filtering strategy. For each candidate lesion trajectory, the trajectory consistency score is obtained by aggregating the lesion consistency scores of each stage. When there is a query time constraint, the correlation evaluation of the query stage constraint is performed based on the stage semantic features of the candidate lesion trajectory to generate a stage consistency score. The trajectory consistency score and the period consistency score are aggregated according to the preset fusion model to form a retrieval score, and the candidate lesion trajectories are ranked accordingly.

2. The method of claim 1, wherein, The generation of the stage semantic set reflecting the changes of lesions across stages specifically includes: Identify relative time descriptions and lesion change descriptions in the current stage of diagnosis and treatment text; Based on the timestamp of the current period, the relative time description is mapped to the target period number on the time axis; When the description of lesion change includes lesion generation or disappearance events, the previous period closest to the current period on the timeline is used as the target period number. Extract the location, direction of change, and magnitude of lesion changes from the description of lesion changes, and encode these three factors together into quantitative semantic features; Structured triples are constructed using the current period number, the target period number, and the quantitative semantic features. All structured triples are then aggregated to form a period semantic set.

3. The method of claim 1, wherein, Phase markers are annotated for each lesion based on sequence information generated from parsed metadata, specifically including: The medical images include CT images, MRI images, and ultrasound images; For CT images, extract the protocol name and scan time from the metadata, determine the phase according to the first preset rule, and mark the phase as arterial phase, portal venous phase, or delayed phase. For MRI images, the sequence name, ImageType and scan time are extracted from the metadata, and the second preset rule is used to determine and mark the phase as T1, T2, DWI or dynamic enhancement period; For ultrasound images or medical images that fail to be determined, the phase identifier is marked as none.

4. The method of claim 3, wherein, The matching degree between lesions and various semantic tags is calculated using physical coordinates, phase identifiers, and quantitative indicators to generate a lesion consistency score, specifically including: Extract the physical coordinates, phase identifiers and quantitative indicators of each lesion in the lesion set, and extract the semantic elements of each semantic tag in the anchor point semantic tag set, and record them as location elements, phase elements and quantitative elements. Calculate the anatomical distance between physical coordinates and location features, map it to a score using a preset attenuation function, and use it as the location matching degree; By comparing the phase identifier with the phase element, the three-state scores of equal, compatible or conflicting are output as the phase matching degree. The quantitative indicators and quantitative elements are input into the Huber difference function to generate a score, which is then used as the quantitative matching degree. The phase matching degree, position matching degree, and quantification matching degree are weighted and aggregated. If the phase identifier is none, the phase matching degree weight is reset to zero and normalized according to the remaining weights, and the lesion consistency score is output.

5. The method of claim 2, wherein, The parsing of the query period constraint specifically includes: Identify absolute period identifiers or relative time descriptions in the query input; For absolute period identifiers, they are parsed into start period number and end period number to form absolute period pairs; For relative time descriptions, they are mapped to the starting period number on the time axis relative to the current timestamp, and all possible period pairs from the starting period number to the current period number are constructed based on a preset check frequency, forming a set of relative period pairs; The union of absolute period pairs and relative period pairs is used as the output query period constraint.

6. The method of claim 5, wherein, The relevance of the query period constraint is evaluated based on the semantic features of the candidate lesion trajectory, generating a period consistency score, specifically including: Traverse each structured triplet in the semantic features of the issue, extract the current issue number and the target issue number to construct issue pairs, and form the actual issue set; The overlap between the actual period set and the query period constraint is calculated and recorded as the period consistency score. The overlap is obtained by dividing the number of elements in the intersection by the number of elements in the union.

7. The method of claim 1, wherein, After sorting the candidate lesion trajectories, a visual search display is also included: Visualize and output the medical images and diagnostic text fragments corresponding to the lesion trajectory in chronological order; The visualization output includes trajectory consistency scores and retrieval scores, and the evolution features of lesion trajectories are compared and contrasted with the semantic features of stages.

8. A cross-modal associated retrieval system for hospital archives, characterized by, A method for implementing cross-modal association retrieval of hospital records as described in any one of claims 1-7, comprising: The multi-source spatiotemporal alignment module is used to acquire medical images of the target object at different times along the same time axis and their associated diagnostic and treatment texts. The lesion quantification and annotation module is used to detect and generate a lesion set from medical images of the same period, calculate the physical coordinates and quantification indicators of each lesion in a unified coordinate system, and annotate the phase identifier of each lesion based on the sequence information generated from the medical image metadata. The text semantic parsing module is used to perform semantic parsing on the diagnosis and treatment text in this period, and generate anchor semantic tags that describe the location, phase or quantitative features of the lesion, as well as a period semantic set that reflects the changes of the lesion across periods. The cross-modal consistency module is used to calculate the matching degree between the physical coordinates, phase identifiers and quantitative indicators of each lesion and the corresponding semantic tags based on the lesion set and anchor point semantic tag set of the same period, and aggregate them to form a lesion consistency score. The lesion trajectory modeling module is used to perform coordinate system registration on medical images of different stages, construct the lesion trajectory of the same lesion on the time axis based on physical coordinates and quantitative indicators, and extract and structure the semantic set of the associated stages to obtain the semantic features of the stages. The query parsing module is used to obtain query input and parse it into a set of query semantic tags and optional query period constraints, and determine the candidate lesion trajectory according to the preset filtering strategy based on the set of query semantic tags. The trajectory scoring module is used to obtain a trajectory consistency score for each candidate lesion trajectory by aggregating the lesion consistency scores of each stage of the lesion. When there is a query time constraint, the module evaluates the relevance of the query stage constraint based on the stage semantic features of the candidate lesion trajectory and generates a stage consistency score. The retrieval and ranking module is used to aggregate trajectory consistency scores and period consistency scores according to a preset fusion model to form a retrieval score, and to rank candidate lesion trajectories accordingly. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. When the processor executes the computer program, it implements the method described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images