Disease monitoring method and system based on medical image data recognition

Abstract

The invention provides a disease monitoring method and system based on medical image data recognition, and relates to the technical field of image processing. The method comprises the following steps: acquiring multi-level prior information related to a target disease from a medical knowledge graph, encoding focus attributes and time sequence attributes in the multi-level prior information into knowledge prior vectors, inputting the knowledge prior vectors into a generator, and enabling the generator to output focus image data based on the knowledge prior vectors; when the lesion image data output by the generator and the knowledge prior vector are subjected to difference evaluation, the discriminator also calls lesion boundaries and staging limitation rules in the medical knowledge graph to perform compliance judgment on the form and staging state of the lesion image data; updating network parameters of the generator based on a difference evaluation and compliance determination result output by the discriminator to obtain target lesion image data; and combining the focus image data with the original medical image, inputting the combined image data into a preset disease monitoring model, and generating a potential focus detection result.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular, to a disease monitoring method and system based on medical image data recognition. Background Art

[0002] At present, medical imaging plays a key role in clinical diagnosis, but there are still several shortcomings: on the one hand, manual reading of large-scale imaging data is time-consuming and laborious, and subjective factors make it difficult to ensure the integrity and consistency of diagnostic reports; on the other hand, image-assisted diagnosis often lacks full use of multi-dimensional prior information of the disease (such as staging rules, anatomical structure dependencies, temporal evolution, etc.), resulting in limited recognition accuracy and interpretability of rare lesions and complex staging. In addition, traditional methods or systems often lack support for automatic detection and structured description of imaging lesions, making it difficult to meet the needs of intelligent diagnosis for accurate recognition of multiple diseases and multiple stages.

[0003] In order to solve the above problems, for example, a Chinese patent with authorization announcement number CN115062165B discloses a medical imaging diagnosis method and device based on a film reading knowledge graph. The method identifies lesions in medical images through the fusion of multiple algorithms (such as image classification, target detection, image segmentation, etc.), and associates them with a pre-built film reading knowledge graph to generate "structured examination findings" containing abnormal image features and lesion location information, and then uses deep learning models and configuration rules to obtain "examination impressions" and "medical imaging diagnosis reports", which improves the efficiency of diagnostic automation and report generation to a certain extent.

[0004] However, the above scheme mainly focuses on using the image features and location information of the knowledge graph when generating diagnostic reports, and the comprehensive use of disease stage evolution rules, lesion temporal properties, GAN (generative adversarial network) and other methods is not sufficient; there is also a lack of in-depth research on lesion image synthesis, multi-round adversarial training, deep constraints, etc., and it is difficult to take into account the fine coupling of morphology and stage priors, and it is impossible to simulate and detect multi-stage lesions or rare sample lesions more accurately. Based on this, it is necessary to further expand and improve on the basis of existing technologies, and comprehensively use knowledge graphs, GANs and lesion morphology stage prior information to more comprehensively improve the intelligent recognition and diagnosis level of medical images in multi-stage scenarios. Summary of the invention

[0005] In view of the deficiencies of the prior art, the embodiments of the present disclosure provide a disease monitoring method and system based on medical imaging data recognition.

[0006] In a first aspect, the present disclosure provides a disease monitoring method based on medical image data recognition, comprising: Obtain multi-level prior information related to the target disease from the medical knowledge graph, the multi-level prior information includes: Lesion attributes, including lesion size, shape, location, and anatomical structure dependency; At least one temporal attribute describing the progression of the target disease; and lesion boundary and staging restriction rules for restricting the lesion morphology and staging of the target disease; Using a generative adversarial network as the lesion generation model, the generative adversarial network includes a generator and a discriminator; Encoding the lesion attributes and time series attributes in the multi-level prior information into a knowledge prior vector and inputting it into the generator, so that the generator outputs lesion image data based on the knowledge prior vector to match the anatomical structure and progression law of the target disease; While the discriminator performs a difference evaluation between the lesion image data output by the generator and the knowledge prior vector, the discriminator also calls the lesion boundary and staging restriction rules in the medical knowledge graph to perform compliance judgment on the morphology and staging status of the lesion image data; Based on the results of the difference evaluation and compliance determination output by the discriminator, the network parameters of the generator are updated through multiple rounds of adversarial training until lesion image data that meets the multi-level prior information requirements is obtained; The lesion image data that meets the multi-level prior information requirements is combined with the original medical image and input into a preset disease monitoring model to generate potential lesion detection results.

[0007] As an optional implementation, the step of causing the generator to output lesion image data based on the knowledge prior vector includes: Numerically processing the size, shape, position and anatomical structure dependency of the lesion attributes to generate a lesion attribute vector, wherein the lesion attribute vector is used to characterize the spatial characteristics and anatomical priors of the target disease lesion; Performing time series feature extraction on the time series attributes describing the progression law of the target disease, and fusing the obtained time series features with the lesion attribute vector to form a time series fusion vector, wherein the time series fusion vector is used to characterize the dynamic changes of the lesion at different stages and progression stages; In the time series fusion vector, based on the critical threshold and stage interval defined by the lesion boundary and the staging restriction rule obtained from the medical knowledge graph, a priori restriction processing is performed to mark and restrict the time series fusion vector in the lesion morphology and staging dimensions; The temporal fusion vector that has been processed with prior constraints is input into the generator, and the lesion spatial features and progression stage information contained in the temporal fusion vector are used to generate lesion image data during the training and reasoning stages.

[0008] As an optional implementation, the difference evaluation between the lesion image data output by the generator and the knowledge prior vector includes: Extracting lesion image features from the lesion image data output by the generator through a feature extraction algorithm to form an image feature vector; Compare the image feature vector with the knowledge prior vector one by one in the spatial attribute dimension, the temporal progression dimension and the stage restriction dimension, and calculate the difference measure respectively; A difference evaluation result is determined according to the difference measurement results of each dimension, and the difference evaluation result is used to indicate the deviation of the lesion image data from the knowledge prior vector.

[0009] As an optional implementation, the compliance determination of the morphology and staging status of the lesion image data includes: Based on the boundary restriction rules obtained from the medical knowledge graph, the edge contour in the lesion image data is matched with a preset lesion morphology threshold, and a morphology non-compliance indication is output when the lesion morphology exceeds the lesion morphology threshold; Based on the staging interval in the time series attribute, the staging status of the lesion image data is compared with the staging restrictions in the medical knowledge graph, and when the staging status exceeds the staging interval, a staging non-compliance indication is output; The results of morphological compliance and stage compliance determination are integrated to form a compliance determination result, and the compliance determination result is used to guide the continuation or correction of the adversarial training process.

[0010] As an optional implementation, the updating of the network parameters of the generator through multiple rounds of adversarial training based on the difference evaluation and compliance determination results output by the discriminator includes: Obtaining a difference loss based on the difference assessment result, obtaining a compliance loss based on the compliance determination result, and fusing the difference loss and the compliance loss with a preset weight to generate a total loss; During the training process, the generator is optimized by gradient back-propagation; When the total loss is lower than a preset threshold or the number of training rounds reaches an upper limit, the updating of the network parameters of the generator is stopped to obtain lesion image data that meets the requirements of the multi-level prior information.

[0011] As an optional implementation, combining the lesion image data satisfying the multi-level prior information requirements with the original medical image includes: Based on the position attributes in the lesion image data, the lesion pixel coordinates are converted and aligned to the corresponding coordinate positions of the original medical image; After the coordinates are aligned, a preset pixel fusion process is performed on the boundary area where the lesion area overlaps with the original medical image, wherein the pixel fusion process includes performing a gradient correction or a seamless cloning operation on the fusion boundary; After completing the pixel fusion processing of the boundary area, the merged image data is output as the synthetic image data.

[0012] As an optional implementation, the lesion image data satisfying the multi-level prior information requirements is combined with the original medical image and input into a preset disease monitoring model to generate potential lesion detection results, including: Loading a preset disease monitoring model and performing image preprocessing operations on the synthetic image data; In the model inference stage, feature extraction and segmentation calculations are performed on the input synthetic image data to generate the location information, size and shape of the potential lesion area; Based on the potential lesion information obtained by model reasoning, a potential lesion detection result is generated; the potential lesion detection result includes lesion spatial coordinates, morphological parameters, and staging information.

[0013] As an optional implementation, it also includes: In response to the staging information, lesion size, and morphological characteristics of the potential lesion detection result not being consistent with the multi-level prior information, recording the inconsistent information as updated data; The updated data is sent to the update interface of the medical knowledge graph to adjust the information of the target disease in the medical knowledge graph.

[0014] In a second aspect, the disclosed embodiment also provides a disease monitoring system based on medical image data recognition, comprising: an acquisition module, a first processing module, a second processing module, and a generation module; The acquisition module is used to obtain multi-level prior information related to the target disease from the medical knowledge graph, and the multi-level prior information includes: Lesion attributes, including lesion size, shape, location, and anatomical structure dependency; At least one temporal attribute describing the progression of the target disease; and lesion boundary and staging restriction rules for restricting the lesion morphology and staging of the target disease; The first processing module is used to use a generative adversarial network as the lesion generation model, and the generative adversarial network includes a generator and a discriminator; Encoding the lesion attributes and time series attributes in the multi-level prior information into a knowledge prior vector and inputting it into the generator, so that the generator outputs lesion image data based on the knowledge prior vector to match the anatomical structure and progression law of the target disease; While the discriminator performs a difference evaluation between the lesion image data output by the generator and the knowledge prior vector, the discriminator also calls the lesion boundary and staging restriction rules in the medical knowledge graph to perform compliance judgment on the morphology and staging status of the lesion image data; The second processing module is used to update the network parameters of the generator through multiple rounds of adversarial training based on the results of the difference evaluation and compliance determination output by the discriminator until the lesion image data that meets the multi-level prior information requirements is obtained; The generation module is used to combine the lesion image data that meets the multi-level prior information requirements with the original medical image, and input it into a preset disease monitoring model to generate potential lesion detection results.

[0015] Compared with the existing technology, this application organically combines the generative adversarial network with multi-level prior information (including lesion morphology, staging restrictions and temporal attributes), and uses multiple rounds of adversarial training and boundary fusion to achieve accurate simulation and detection of rare staging and multi-morphological lesions. It also forms a closed loop through dynamic feedback between monitoring results and knowledge graphs, significantly improving the accuracy and interpretability of lesion synthesis and identification, thereby better meeting the clinical needs of medical imaging in multi-stage diagnosis scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a disease monitoring method based on medical imaging data recognition provided by an embodiment of the present disclosure; Figure 2 A flowchart of a method for enabling a generator to output lesion image data based on a knowledge prior vector provided by an embodiment of the present disclosure; Figure 3 A schematic diagram of a disease monitoring system based on medical imaging data recognition provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0018] See also Figure 1 FIG. 1 is a flowchart of a disease monitoring method based on medical image data recognition provided by an embodiment of the present disclosure, the method comprising steps S101 to S106, wherein: S101: Acquire multi-level prior information related to a target disease from a medical knowledge graph, the multi-level prior information including: lesion attributes, including lesion size, morphology, location, and anatomical structure dependency; at least one temporal attribute describing the progression law of the target disease; and lesion boundary and staging restriction rules for restricting the lesion morphology and staging of the target disease; S102: Using a generative adversarial network as the lesion generation model, the generative adversarial network comprising a generator and a discriminator; S103: Encoding the lesion attributes and time series attributes in the multi-level prior information into a knowledge prior vector and inputting the knowledge prior vector into the generator, so that the generator outputs lesion image data based on the knowledge prior vector to match the anatomical structure and progression law of the target disease; S104: while the discriminator performs a difference evaluation between the lesion image data output by the generator and the knowledge prior vector, the discriminator also calls the lesion boundary and staging restriction rules in the medical knowledge graph to perform compliance determination on the morphology and staging status of the lesion image data; S105: Based on the results of the difference evaluation and compliance determination output by the discriminator, the network parameters of the generator are updated through multiple rounds of adversarial training until lesion image data that meets the multi-level prior information requirements is obtained; S106: After combining the lesion image data that meets the multi-level prior information requirements with the original medical image, the data is input into a preset disease monitoring model to generate potential lesion detection results.

[0019] The present invention obtains multi-level prior information related to the target disease from a medical knowledge graph; uses a generative adversarial network (GAN) as a lesion generation model; encodes the multi-level prior information (lesion attributes and time series attributes) into a knowledge prior vector and inputs it into a generator; while the discriminator performs a difference evaluation between the lesion image data output by the generator and the knowledge prior vector, it calls the lesion boundary and staging restriction rules in the medical knowledge graph to make a compliance judgment; based on the difference evaluation and compliance judgment results, multiple rounds of adversarial training are performed to update the generator network parameters until the lesion image data that meets the requirements of the multi-level prior information is obtained; the lesion image data that meets the requirements of the multi-level prior information is combined with the original medical image, and then input into the disease monitoring model to generate potential lesion detection results.

[0020] Regarding S101 above: In the present invention, the medical knowledge graph stores the prior knowledge of the target disease in multiple dimensions such as anatomical structure, lesion morphology, temporal progression, and staging restrictions; by extracting this prior information, the subsequent lesion generation and detection process can be closer to the real clinical scenario, improving the reliability of lesion image synthesis and detection results.

[0021] In specific implementation, the medical knowledge graph can be constructed from multiple sources, such as clinical medical records, literature, and medical imaging databases. Through natural language processing and relationship extraction technology, information such as disease name, lesion location, anatomical structure dependency, and staging criteria are stored in the form of a graph structure. For target diseases (such as lung tumors, liver tumors, etc.), lesion attributes (size, morphology, location, anatomical structure dependency), time series attributes (reflecting the progression of the disease over time, such as staging, clinical course, etc.), and lesion boundaries and staging restriction rules (used to limit the lesion morphological boundaries, the range of morphological changes corresponding to different stages, etc.) can be queried in the medical knowledge graph.

[0022] After obtaining the relevant nodes and associated information of the target disease, the lesion attributes, time series attributes and boundary / stage restriction rules are read and preprocessed respectively. For example: Lesion attributes: record the maximum / minimum size of the lesion, common shape type (nodular, mass-like, etc.), specific anatomical location, etc.; Time series attributes: including typical manifestations of the disease at different stages or progression stages and time intervals; Lesion boundary and stage restriction rules: including the numerical threshold regulations for the lesion morphological boundary, the constraints on the size or shape characteristics of lesions of different stages, etc.

[0023] For example, clinical medical records, medical imaging databases, and literature data can be collected to store lesion characteristics (size, morphology, location, staging criteria, etc.) related to the target disease (such as lung cancer, liver cancer) in a node-relationship form in a graph database such as Neo4j.

[0024] For example, if the target disease is "lung cancer", the system can retrieve the typical size range of the disease in stages I-IV (such as a few millimeters to a few centimeters), common shape types (nodular, patchy, etc.), and corresponding lesion boundary restriction rules (such as edge smoothness, burrs, etc.).

[0025] Regarding S102 above: This paper selects the generative adversarial network (GAN) as the lesion generation model. Using the adversarial training mechanism of GAN, the lesion morphology and staging characteristics are learned from the distribution of prior information and a small number of real samples, and then the lesion image data that is closer to clinical reality is synthesized; it provides a network framework foundation for the compliance judgment and adversarial training of the subsequent discriminator. GAN consists of two parts: the generator and the discriminator, where: A generator, used for receiving inputs such as a priori vectors and outputting lesion image data; The discriminator is used to evaluate the lesion image output by the generator and determine its authenticity or compliance based on prior knowledge.

[0026] In a specific implementation, the generator and discriminator can be implemented using a convolutional neural network (CNN) or other deep network structures, and the initial parameters can be randomly initialized or transferred from a pre-trained model.

[0027] In addition, if more complex time series features are involved, additional time series modeling modules (such as LSTM, Transformer, etc.) can be introduced into the generator or discriminator to adapt to the progress rules described in the multi-level prior information.

[0028] For example, mainstream GAN structures such as DCGAN and WGAN can be selected, and both the generator and the discriminator are based on convolutional neural network (CNN).

[0029] Regarding S103 above: The present invention converts discrete or continuous information such as lesion size, morphology, location, and temporal progression into a network-readable vector format; introduces prior rules such as lesion boundaries and staging restrictions, and imposes necessary upper and lower limit constraints on the vector, so that the generator complies with anatomical and staging priors when synthesizing lesions.

[0030] In the specific implementation, the lesion attributes (size, morphology, location, anatomical structure dependency, etc.) are first characterized, for example, the morphology is represented as a shape vector of several dimensions, and the location is represented as coordinates or mask information to form a lesion attribute vector. In the time series attributes, the information of different stages or phases is serialized, for example, the features corresponding to stage I, stage II, and stage III are quantified as discrete or continuous variables; if there is time series data (such as follow-up time interval, disease course evolution law, etc.), time series feature extraction methods (such as multi-scale time series embedding) can be used to generate time series vectors.

[0031] Furthermore, after fusing the lesion attribute vector with the time series vector, a preliminary time series fusion vector is obtained.

[0032] The present disclosure also obtains threshold or rule information corresponding to lesion boundaries and staging restriction rules from the medical knowledge graph, and performs a priori restriction processing on the time series fusion vector.

[0033] Exemplarily, if the staging restrictions specify the maximum morphological range, minimum aspect ratio, etc. of the lesion, corresponding upper and lower limit annotations are set in the fusion vector.

[0034] Finally, the fusion vector after prior restriction processing is input into the generator, so that the generator can use the prior vector to synthesize lesion images during the training and inference stages, thereby ensuring that the generated lesion images are more in line with actual clinical needs and progression laws.

[0035] Exemplarily, when lesion attributes are vectorized, the lesion size can be quantified into a one-dimensional or two-dimensional value, the lesion morphology can be encoded by shape (such as spherical, elliptical, lobed, etc.), the lesion position can be encoded by coordinates or mask information, and the dependency relationship can be attached in the form of One-Hot or embedded vector.

[0036] For example, when performing time series attribute fusion, if the target disease has multiple CT scan time points, the lesion volume growth rate information or morphological change rate of each stage can be extracted; and the typical morphological differences in different stages can be encoded.

[0037] For example, when performing a priori restriction processing, the time series fusion vector can be calibrated according to the "stage restriction rules" recorded in the medical knowledge graph. For example, when the lesion is in stage III, its diameter is required to be ≥X mm; if it does not meet the requirements, the dimension is corrected to the compliance range.

[0038] As an alternative implementation, see Figure 2 , Figure 2 A flowchart of a method for enabling a generator to output lesion image data based on a knowledge priori vector provided by an embodiment of the present disclosure includes steps S201 to S204, wherein: S201: numerically processing the size, shape, position and anatomical structure dependency of the lesion attributes to generate a lesion attribute vector, wherein the lesion attribute vector is used to characterize the spatial characteristics and anatomical priors of the target disease lesion; S202: extracting time series features from the time series attributes describing the progression law of the target disease, and fusing the obtained time series features with the lesion attribute vector to form a time series fusion vector, wherein the time series fusion vector is used to characterize the dynamic changes of the lesion at different stages and progression stages; S203: performing a priori restriction processing on the time series fusion vector based on the critical threshold and stage interval defined by the lesion boundary and staging restriction rule obtained from the medical knowledge graph, so as to mark and restrict the time series fusion vector in the lesion morphology and staging dimensions; S204: Inputting the time series fusion vector that has undergone a priori restriction processing into the generator, and generating lesion image data in the training and reasoning stages using the lesion spatial features and progression stage information contained in the time series fusion vector.

[0039] In order to enable the generator to more accurately output lesion image data that meets clinical needs, it is necessary to digitize and fuse the lesion attributes and time series attributes, and implement prior restriction processing based on the lesion boundaries and staging restriction rules in the medical knowledge graph.

[0040] Regarding the above S201: The discrete or continuous information such as lesion size, shape, location and anatomical structure dependency is converted into numerical vectors that can be read by the generator, giving the lesion the ability to express "spatial characteristics" and "anatomical priors".

[0041] In a specific implementation, one-dimensional or multi-dimensional numerical quantities can be defined for the maximum diameter, minimum diameter, or major and minor diameters of the lesion, such as 10 mm to 15 mm to represent the size range; if common morphologies of lesions include spherical, elliptical, lobed, etc., the morphological type can be recorded using One-Hot or embedding vectors; indexing can be performed using three-dimensional (x, y, z) coordinates or based on anatomical sections (such as lung lobes, liver segments), thereby recording the position of the lesion in the original image coordinate system or anatomical structure.

[0042] In addition, for anatomical structure dependencies, if the medical knowledge graph indicates that "lesions mostly appear near the vascular structure around a certain lung segment", the relationship can be assigned a specific identifier or numerical value to indicate the closeness of its relationship with the surrounding tissues.

[0043] For example, if the target disease is lung cancer: the size quantization dimension (14.0, 9.5) indicates that the long diameter is 14 mm and the short diameter is 9.5 mm; the morphological code (0, 1, 0) indicates an ellipse (One-Hot vector); the position coordinates (45, 60, 32) indicate the three-dimensional coordinates of the center of the lesion in a certain lobe; the anatomical structure dependency can be additionally represented by 1 to indicate "close to the hilar vessels". Finally, these numerical information are combined into a lesion attribute vector, which can be further integrated with the time series information in subsequent steps.

[0044] Regarding the above S202: The time series feature extraction and fusion of temporal attributes enable the generator to reflect the dynamic evolution of the disease over time or with the stage (such as how the lesion becomes larger or changes in morphology from stage I to stage II), thereby having multi-stage and a series of time-varying characteristics when synthesizing lesion images.

[0045] In specific implementation, the typical manifestations (morphological change rate, tumor growth rate, etc.) of the target disease in different stages (I, II, III, IV) or clinical progression stages can be queried from the medical knowledge graph. If there are multi-stage imaging data or lesion evolution rates recorded in clinical literature, multi-scale temporal embedding, LSTM or other temporal models can be used to extract feature vectors that reflect the trend of lesion changes.

[0046] The time series features are concatenated or weightedly combined with the aforementioned lesion attribute vectors dimension by dimension to form a time series fusion vector, which represents the comprehensive attributes (size, shape, speed, stage, etc.) that the lesion should have at the current moment (or stage).

[0047] For example, for the typical growth rate (0.5, 0.7, 1.2) (unit: mm / week) from stage I to stage III of lung cancer, it can be used as a time series feature and the lesion attribute vector (14.0, 9.5) (size) to form a three-dimensional feature such as (14.0, 9.5, 0.7); If the disease has multiple lobes in stage III morphology, the morphological type code (lobed) can be additionally annotated in the temporal fusion vector to make the generator present a more complex outline during synthesis.

[0048] Regarding the above S203: The present invention utilizes critical thresholds, stage intervals, etc. defined in medical atlases to perform range detection and correction on time series fusion vectors, thereby ensuring the "rationality" of the lesions in morphology and staging, and avoiding the generation of unrealistic lesion images.

[0049] In the specific implementation, for boundary restrictions, if the smoothness of the lesion edge is specified in the atlas as 0.1, 0.3 (indicating smoothness to the degree of lobulation), when the corresponding dimension in the fusion vector exceeds 0.3, it will be truncated or regressed to 0.3; for stage restrictions, if the stage is stage II and the time series characteristics indicate that the tumor diameter should be between 8 mm and 20 mm, then when the fusion vector shows a diameter of <8 mm or >20 mm, it will be corrected.

[0050] For example, when the stage is in stage III, but the dimension of "morphological concavity" in the fusion vector is too small (too smooth), the dimension value is automatically increased according to the restriction rule to reflect the lobulation or spiculation of the stage III lesion; When the stage is in stage I, but the fusion vector indicates that the long diameter of the tumor is greater than 30 mm, it is retracted to less than 8 mm to meet the size limit of stage I.

[0051] In this way, after completing the prior restriction processing, the obtained time series fusion vector is more consistent with the medical knowledge graph prior in terms of lesion morphology and staging, providing reliable input for subsequent synthesis.

[0052] Regarding the above S204: The present invention uses the above-mentioned fused vectors processed with prior constraints as the main input of the generator in the training and reasoning stages, so that the synthesized lesion images are closer to the actual clinical manifestations.

[0053] In the specific implementation, for the training stage: in the adversarial training cycle, one or a batch of time-series fusion vectors are extracted from the atlas prior in each round; the generator outputs the lesion image based on the fusion vector; the discriminator performs difference evaluation and compliance judgment, and feeds back the difference loss and compliance loss; iterates continuously until the generator learns to produce "high compliance and low difference" lesion images.

[0054] For the reasoning stage: given the target stage or morphological requirement, the input vector is obtained through the same fusion and prior restriction processing; the vector is input into the trained generator, which outputs the corresponding lesion image; the output result can be combined with the real image or subsequent detection model to realize the simulation and analysis of rare morphology or multi-stage lesions.

[0055] For example, if clinical simulation of liver cancer lesion morphology at stage III is needed, the size threshold and lobation degree information at stage III can be obtained from the atlas first, integrated into the lesion attribute vector and the time series attribute vector, and then input into the generator after prior restrictions.

[0056] During the inference stage, the generator can output lesion images that meet the typical manifestations of stage III (such as a diameter of 25 mm and multi-lobed edges) for testing or comparison by physicians or AI detection models.

[0057] In this way, during the training phase, the fused and prior-restricted vectors can be used to continuously optimize the generator so that it approaches multi-level prior information constraints; during the inference phase, the same vector can be input to generate lesion images that meet the target staging or morphological characteristics, thereby achieving the simulation and detection requirements of rare lesions or multi-stage lesions in both stages, ensuring the inherent consistency between morphology and staging, and better meeting the clinical accuracy and lesion diversity requirements.

[0058] Regarding S104 above: The present invention measures in real time the deviation between the lesion image output by the generator and the prior vector in the spatial, temporal and staging dimensions; by calling the lesion boundary and staging rules in the atlas, it judges whether the morphology and staging "violate clinical priors" and forms "compliant" or "non-compliant" feedback.

[0059] In the specific implementation, after the generator outputs the lesion image data, the discriminator extracts features from it, and calculates the difference metrics (such as L1 distance, cosine similarity, etc.) between the extracted image features and the knowledge prior vector in the spatial attributes, temporal progression, and stage restriction dimensions, to form a difference evaluation result, indicating the degree of deviation of the lesion image from the prior vector.

[0060] In addition, while performing the difference assessment, the discriminator further calls the lesion boundary and stage restriction rules recorded in the atlas to perform morphological and stage compliance checks.

[0061] For example, if the morphology exceeds the allowed range of the lesion boundary, the discriminator outputs that the morphology is non-compliant; if the staging status is not within the a priori specified staging interval, the discriminator outputs that the staging is non-compliant.

[0062] Finally, the morphological compliance and stage compliance results are integrated into the compliance determination result.

[0063] Exemplarily, the discriminator extracts features from the lesion image data through CNN and other methods to obtain an image feature vector, which is compared one by one with the knowledge prior vector in spatial dimensions (such as morphological edge smoothness) and temporal dimensions (such as growth rate), and metrics such as cosine similarity or L1 distance are calculated.

[0064] For example, if the edge of the lesion is too sharp or the concave and convex exceeds the stage limit threshold in the atlas, the morphology is judged to be non-compliant; if the temporal stage interval does not match (for example, the stage information should be stage II but meets the stage IV morphology), the stage is judged to be non-compliant. The discriminator outputs the morphology and stage determination results in a comprehensive manner.

[0065] As an optional implementation, the difference evaluation between the lesion image data output by the generator and the knowledge prior vector includes: Extracting lesion image features from the lesion image data output by the generator through a feature extraction algorithm to form an image feature vector; Compare the image feature vector with the knowledge prior vector one by one in the spatial attribute dimension, the temporal progression dimension and the stage restriction dimension, and calculate the difference measure respectively; A difference evaluation result is determined according to the difference measurement results of each dimension, and the difference evaluation result is used to indicate the deviation of the lesion image data from the knowledge prior vector.

[0066] In the adversarial training framework of the present invention, the discriminator needs to consider the matching degree between the lesion image data and the knowledge prior vector in multiple dimensions (such as spatial attributes, temporal progression, and stage restrictions) at the same time. Through difference evaluation, the deviation between the lesion image and the prior vector can be quantified, and the difference evaluation results can be further generated to provide a basis for subsequent compliance judgment or loss function fusion.

[0067] The lesion image data output by the generator is converted into a numerical feature representation to facilitate dimension-by-dimension comparison with the prior vector, and then the differences between the lesion and the prior vector in terms of morphology, size, timing, etc. are evaluated.

[0068] In specific implementations, a convolutional neural network (CNN) can be used to perform multi-layer convolution and pooling operations on lesion image data to obtain feature maps at one or more levels. A fixed-length vector, called an image feature vector, can be output at the last layer or the last few layers. In addition, if richer morphological information is needed, a vector can be obtained by fully connecting the local texture features extracted at the second-to-last layer, or by using methods such as global average pooling (GAP) to ensure the controllability of the vector length.

[0069] In the specific implementation, if the generator output contains the complete lesion and background, the lesion area can be preliminarily segmented or cropped first to reduce the interference of irrelevant background on the features; if full-image feature extraction is used, it is necessary to ensure that CNN can focus on the main lesion area and reduce background noise interference through attention mechanisms and other means.

[0070] For example, if the size of the lesion image is 128×128 pixels, a lightweight CNN structure (such as 4-layer convolution) can be used to extract a 64-dimensional lesion image feature vector F img =[ f 1 , f 2 ,…, f d ] ;in, is the high-dimensional vector obtained after feature extraction algorithms such as CNN; d is the dimension of the feature vector. If CNN outputs 64 dimensions, then d=64; if a more complex network outputs 128 dimensions, then d=128; Represents the i-th eigenvalue, such as the morphological information or texture information learned by the deep network.

[0071] If more spatial resolution information is needed in the implementation, a layer of SPP (spatial pyramid pooling) can be added after CNN to perform multi-scale feature fusion.

[0072] The present invention calculates the difference measure in the spatial attribute dimension, the temporal progression dimension, and the stage restriction dimension respectively, to avoid mixing all information into a whole loss and ignoring key details. By comparing dimension by dimension, the deviation degree of the lesion in terms of shape size, edge smoothness, temporal growth rate, stage sign, etc. can be precisely measured.

[0073] In a specific implementation, if the dimension of the prior vector contains lesion spatial attributes (such as size, morphological coding), temporal attributes (such as growth rate or stage label), stage restriction, etc., then the corresponding mapping or coding should also be included in the image feature vector; If the number of dimensions is inconsistent, an index or sub-vector correspondence can be established between the prior vector and the image feature vector, and expansion / dimensionality reduction can be performed at the CNN output end or the prior vector end.

[0074] Additionally, for the difference metric calculation: Morphological / spatial attribute dimension: For example, using L2 distance (L2 Norm) or cosine similarity to measure the similarity between lesion morphological vectors; Time series dimension: For example, compare the "growth rate" or "stage" mapping value and perform subtraction or similarity calculation with the corresponding value learned from the image features; Staging restriction dimension: If the staging restriction stipulates that the lesion must not exceed a certain morphological threshold, the corresponding staging code of the image feature vector is compared with the prior staging value, and the difference is marked if it exceeds the interval.

[0075] Furthermore, the difference measurement results of each dimension are combined to generate a difference evaluation result, which is used to indicate the overall deviation, or as a reverse optimization signal output to the generator as part of the discriminator in the adversarial training process.

[0076] In the specific implementation, different weights can be assigned to the three dimensions of morphology, timing, and stage restrictions, and the obtained difference measurement values ​​can be multiplied by the corresponding weights and then summed or averaged to generate a total difference evaluation score. ,The larger the value, the higher the lesion deviation, and the smaller the value, the more consistent with the prior.

[0077] In adversarial training, Combined with other losses (such as morphological / periodic non-compliance in compliance determination) for the generator to adjust network parameters during gradient backpropagation; In the inference phase, The output is given to doctors or subsequent detection models to indicate the degree of conformity of the current synthetic lesion with the prior dimension.

[0078] For example, in adversarial training, if A larger value indicates that the lesion image is seriously inconsistent with the prior, and a higher difference loss will be generated in the training loss function for backpropagation to the generator; if It is small, indicating that the morphology and temporal characteristics of the lesion are highly consistent with the prior, and its penalty intensity can be reduced during training.

[0079] For example, in the inference scenario, when the system is only inferring or simulating the lesion, by calculating It can evaluate whether the current synthetic lesion exceeds the a priori range; if it deviates too much, it reminds the user to adjust the staging settings or attribute parameters. Compare with the subsequent test results to form indicators of "lesion rationality" or "simulation case reliability".

[0080] For example, if the typical diameter interval of stage II lung cancer is specified as [10mm, 20mm], and the growth rate is less than or equal to 1.0 mm / week, the generator outputs the lesion image feature vector F img =[15mm,0.8,…] , falls into the interval both in space and time; the difference measure is small, and the judgment of the lesion is consistent with the prior.

[0081] For example, if the lesion is designated as stage III a priori, lobulation or burr is allowed, but the maximum diameter should not be less than 25 mm; the maximum diameter of the lesion image output by the generator is only 18 mm, so the difference in the staging restriction dimension is large, resulting in The training process will increase the penalty on the generator, prompting it to generate lesions with larger diameters or more consistent with stage III morphology in the next iteration.

[0082] For example, if the difference measurement values ​​of each dimension are: spatial difference = 0.15, temporal difference = 0.02, stage difference = 0.3, and the weights are 0.4, 0.3, 0.3, then: when When <0.10, it can be considered that the lesion image is very consistent with the prior. When it is greater than 0.30, the deviation is relatively large and further correction is required by the generator.

[0083] For example, the prior vector is also d-dimensional, or can be split into Corresponding sub-vectors, cosine similarity It can be expressed as: in," " represents the vector dot multiplication operation, and Represents the vector norm.

[0084] In this way, through the above-mentioned process of feature extraction, dimension-by-dimension comparison, and difference evaluation results, the present disclosure can carefully measure the degree of fit between lesion image data and prior vectors in adversarial training or reasoning. After the difference evaluation is combined with the subsequent morphological / staging compliance judgment, more targeted feedback can be provided to the generator, so that it can continuously approach the actual clinical needs in the synthesis of lesion spatial features and temporal staging features, thereby achieving the effect of improving the credibility of lesion image synthesis and assisting the training of disease monitoring models.

[0085] As an optional implementation, the compliance determination of the morphology and staging status of the lesion image data includes: Based on the boundary restriction rules obtained from the medical knowledge graph, the edge contour in the lesion image data is matched with a preset lesion morphology threshold, and a morphology non-compliance indication is output when the lesion morphology exceeds the lesion morphology threshold; Based on the staging interval in the time series attribute, the staging status of the lesion image data is compared with the staging restrictions in the medical knowledge graph, and when the staging status exceeds the staging interval, a staging non-compliance indication is output; The results of morphological compliance and stage compliance determination are integrated to form a compliance determination result, and the compliance determination result is used to guide the continuation or correction of the adversarial training process.

[0086] Among them, "staging restriction" in this application text refers to the comprehensive restriction rules for lesion staging, which may include constraints on factors such as size, shape, and growth rate; "staging range" focuses more on numerical ranges (such as diameter, upper and lower limits of growth rate).

[0087] In the discriminator of the present invention, in addition to the difference assessment, it further determines whether the "lesion morphology" and "stage status" exceed the limit range defined in the medical knowledge graph. If the morphology is not compliant or the stage is not compliant, the discriminator will output the corresponding non-compliance indication to help the adversarial training process make targeted corrections to the generator.

[0088] In specific implementations, the medical knowledge graph usually records the morphological features and boundary limits of the target disease at different stages. For example, if a certain stage of lung cancer lesions has a threshold interval in terms of contour smoothness, burr coefficient, concave and convex degree, etc., the present disclosure will extract the interval and match it in the discriminator.

[0089] In the medical knowledge graph, for the target disease's stage I, stage II, stage III, stage IV, or different stages of progression, the corresponding size, shape, growth rate, and other intervals are often defined. For example: Stage I tumors are usually less than 20 mm in diameter; In stage III, the morphology is more lobulated or invasive, and the size is greater than 25 mm.

[0090] These staging intervals can be recorded in the medical knowledge graph through numerical upper and lower limits or specific annotations.

[0091] Furthermore, when the discriminator extracts features from the lesion image, it can additionally detect or estimate the edge contour of the lesion, such as morphological indicators such as smoothness, number of lobed protrusions, and degree of burrs.

[0092] When extracting the features of the lesion image, the discriminator can also infer or estimate its corresponding "staging tendency", for example, by comprehensively analyzing the morphology, size, texture and other information, to obtain a staging judgment value; If the inferred staging status exceeds the specified range, such as "should be in stage II but shows a size or invasiveness equivalent to stage IV", an "non-compliant staging" indication will be output.

[0093] If the morphological index exceeds or falls below the upper and lower thresholds in the medical knowledge graph, it will be judged as "morphological non-compliance" and a corresponding indication will be output; For example, if the stage II requirement is that the burr degree does not exceed a certain value, once the value is exceeded, a morphological non-compliance prompt can be output.

[0094] The discriminator will combine the results of the morphological compliance and the results of the installment compliance to form an overall "compliance determination result". If both are compliant, a "compliant" indication can be output; if any dimension is non-compliant, it can be recorded as "non-compliant" with the specific reasons for the morphological or installment violations.

[0095] The compliance judgment result can be used together with the "difference metric" generated by the difference evaluation to act on the generator during the adversarial training process: if the judgment result shows that the morphology or staging is not compliant, the generator will correct the lesion sample by increasing the loss term in the next round of training; if it is compliant, it means that the lesion basically meets the graph prior and no further correction is required in terms of morphology or staging.

[0096] For example, if the burr degree range for a certain stage of lung cancer is 0.1 to 0.3, and the discriminator detects that the lesion burr coefficient is 0.45, which exceeds the upper limit of 0.3, then it outputs "morphological non-compliance indication". During training, a greater loss will be imposed on the generator to reduce the burr degree in the next round of synthesis.

[0097] For example, if the stage II tumor diameter range is 10-20 mm, and the inferred diameter in the lesion image is 25 mm, it is judged as "non-compliant with the stage". Subsequent adversarial training will increase the corresponding penalty term so that the generator can reduce the diameter to within 10-20 mm during synthesis.

[0098] For example, if the lesion is compliant in the morphological dimension but not in the staging dimension, the final compliance judgment result is recorded as "non-compliant", and the generator is updated with corresponding gradients during training; if both are compliant, a compliance indication is output, and the difference loss is relatively low.

[0099] In this way, compared with a simple difference assessment, this implementation method can make a more fine-grained judgment through "morphological compliance" and "staging compliance", which can significantly reduce the number of lesion samples synthesized by the generator that do not conform to clinical common sense, thereby improving the credibility of the final generated results. Regardless of whether the lesion is in the early stage, mid-stage or late stage, its size and morphological restrictions can be flexibly determined according to the staging interval, which is applicable to a variety of malignant tumors or other disease types.

[0100] It should be emphasized that the present disclosure is not limited to malignant tumors, but can also be extended to benign lesions or pathological scenarios of other organs; it is only necessary to add corresponding morphological rules or staging specifications in the medical knowledge graph.

[0101] Regarding the above S105: The present invention integrates the "difference assessment results" and the "compliance determination results" into a total loss to achieve joint optimization of the lesion images in terms of spatial morphology and clinical staging during adversarial training; and improves the "clinical acceptability" of the generator output through multiple rounds of iterations to reduce the deviation from prior knowledge.

[0102] In the specific implementation, the difference loss is obtained according to the difference evaluation result, and the compliance loss is obtained according to the compliance judgment result, and they are fused with a preset weight to form a total loss. The generator can be back-propagated with gradients through the standard GAN training process; after each round of training, the discriminator performs difference evaluation and compliance judgment on the newly output lesion image. This cycle is repeated for multiple rounds until the total loss is lower than the threshold or the number of training rounds reaches the upper limit. Finally, the lesion image data that can accurately generate anatomical structure and staging prior requirements is obtained.

[0103] Among them, for the difference loss, in adversarial training, the discriminator first evaluates the difference between the lesion image data and the prior vector, and calculates the deviation of the lesion in spatial morphology, temporal characteristics and staging dimensions. This deviation can be directly regarded as "difference loss", or appropriate weighting / normalization processing can be performed on this basis to form the final value used for gradient back propagation.

[0104] For compliance loss, this disclosure records each "non-compliant" event as a corresponding penalty amount, called "compliance loss". For example, if the degree of morphological transgression is large, a higher weight can be given in the loss; if the gap between the installment and the prior is very outrageous, the loss can also be increased.

[0105] When the lesion satisfies the morphology and staging priors, the compliance loss can be recorded as 0 or a smaller value.

[0106] Exemplarily, the difference loss uses L2 distance, and the compliance loss uses "morphological over-limit penalty" or "stage over-limit penalty", which are combined into the total loss at a ratio of 0.5:0.5; when the total loss is lower than a certain threshold (such as 0.01) or the number of training rounds reaches a preset upper limit (such as 200 epochs), the training can be stopped; at this time, the generator can better synthesize lesion images that conform to the prior rules.

[0107] As an optional implementation method, in actual application, a phased strategy can also be adopted: In the early stage of training, increase the difference loss ratio to let the generator learn the basic form first; In the later stage, the compliance loss ratio is increased to make the generator more accurate in detail classification and morphological features.

[0108] Regarding S106 above: The present invention fuses the synthesized lesions with the original medical images to form "simulated cases" or "images with lesion features" to input into subsequent disease monitoring models, simulate scarce lesion data in clinical scenarios, and enhance the detection model's ability to recognize multi-stage or rare morphological lesions.

[0109] In the specific implementation, after obtaining the lesion image data that meets the multi-level prior information requirements, it is combined with the original medical image (such as through coordinate mapping, seamless cloning, gradient correction and other fusion algorithms) to generate synthetic image data. The synthetic image data not only retains the original image context, but also adds lesion information that meets the time sequence and stage priors at the target location.

[0110] The synthetic image data is input into a disease monitoring model for inference. For example, a deep learning detection / segmentation model is used to infer the synthetic image to obtain potential lesion areas and possible staging information, morphological parameters, etc.

[0111] For example, the output result may be in the form of: Lesion coordinates or mask; Lesion morphological indicators (long diameter, short diameter, density, etc.); Staging identification (such as stage I, stage II, etc.) or benign or malignant judgment (such as benign nodules or malignant tumors).

[0112] In addition, the result can be compared or updated again with the medical knowledge graph to form a closed-loop knowledge and model iteration.

[0113] For example, networks such as UNet, Mask R-CNN, and YOLO can be used to automatically detect / segment synthetic images, and output potential lesion location, size, shape, or staging information. If the identification is consistent with the staging recorded in the medical knowledge graph, the model detection is judged to be correct; if the detection deviation is too large, it can be fed back to the medical knowledge graph or GAN model for correction.

[0114] In this way, the present invention can make full use of the multi-level prior information (including temporal staging) recorded in the medical knowledge graph in the lesion synthesis stage, and use a combination of difference evaluation and compliance judgment in the adversarial training stage to make the generated lesions better meet the actual clinical needs.

[0115] After the synthetic images are input into the disease monitoring model, in the case of rare lesions or scarce staging samples, they can supplement the shortage of real samples, thereby improving the accuracy of disease detection or staging identification.

[0116] As an optional implementation, combining the lesion image data satisfying the multi-level prior information requirements with the original medical image includes: Based on the position attributes in the lesion image data, the lesion pixel coordinates are converted and aligned to the corresponding coordinate positions of the original medical image; After the coordinates are aligned, a preset pixel fusion process is performed on the boundary area where the lesion area overlaps with the original medical image, wherein the pixel fusion process includes performing a gradient correction or a seamless cloning operation on the fusion boundary; After completing the pixel fusion processing of the boundary area, the merged image data is output as synthetic image data to reflect the position and morphological information of the lesion in the original medical image.

[0117] In specific implementation, the lesion image data usually includes position attributes or spatial coordinate annotations, reflecting the reference location of the lesion before synthesis. If the location is inconsistent with the coordinate system of the original medical image, it needs to be converted first. The specific approach can be divided into the following steps: In the generation stage, if the center point (or key point) and size information of the lesion image have been recorded, coordinate alignment can be achieved through simple translation, scaling, or 2D / 3D rotation; If the lesion morphology is complex, a small-scale interpolation algorithm (such as nearest neighbor or bilinear interpolation) can be introduced during coordinate transformation to ensure the consistency of image resolution and coordinate system.

[0118] Through the above-mentioned coordinate mapping operation, the lesion image can be seamlessly placed at the corresponding anatomical position of the original image, thereby ensuring the rationality of the final synthesized image in terms of anatomical structure.

[0119] For example, in the lung scenario, the lesion is placed in the specified lung segment; in the liver scenario, the coordinate range of the liver segment is specified.

[0120] Furthermore, when the lesion image coordinates are aligned with the original medical image coordinates, pixel-level differences may appear at the border or in the overlapping area. The present disclosure solves the problem of obvious splicing marks or abrupt transitions that may occur by: Method 1: Gradient correction. If there is a jump in grayscale or brightness in the boundary area, gradient smoothing can be performed to make the pixels at the edge of the lesion gradually transition with the surrounding tissue. This can be achieved by calculating the gradient field of the boundary area and performing low-intensity interpolation or blending.

[0121] Method 2: Seamless cloning. If a higher quality fusion with better visual continuity is required, seamless cloning techniques such as Poisson Blending can be used to clone the lesion area as the foreground into the background image, so that it can be naturally connected with the surrounding tissue in terms of color and brightness. This method is suitable for scenes with large differences in color or grayscale gradients.

[0122] Method 3: After the above fusion steps are completed, the processed synthetic image data can be output as the merged image data, which becomes the "synthetic image data". The synthetic image not only retains the background and anatomical structure information of the original image, but also presents the lesion image at the corresponding position, ensuring smooth connection and visual consistency in the edge area.

[0123] Furthermore, after coordinate alignment and boundary fusion, the final synthetic image can not only accurately display the expected anatomical position of the lesion, but also maintain the integrity of the lesion morphology and the original image background.

[0124] Exemplarily, the synthetic image data is applicable to the following scenarios: Disease monitoring model training: Use the generated simulated cases to expand the training set and improve the recognition ability of multi-stage or rare lesion samples; Clinical assistance: Physicians can observe the interaction between lesions and surrounding tissues in synthetic images, or compare them with real cases to assist diagnosis or teaching; Algorithm verification: In a research environment, different fusion algorithms (such as gradient correction and seamless cloning) are used to compare the quality of the synthesized image and the effect of improving detection accuracy.

[0125] In this way, through the coordination of the three steps of lesion pixel coordinate conversion, boundary fusion processing and synthetic image output in this embodiment, the final synthetic image can conform to clinical logic in form and position, maintain sufficient pixel continuity, and avoid obvious stitching marks or unnatural transitions.

[0126] As an optional implementation, the lesion image data satisfying the multi-level prior information requirements is combined with the original medical image and input into a preset disease monitoring model to generate potential lesion detection results, including: Loading a preset disease monitoring model and performing image preprocessing operations on the synthetic image data; In the model inference stage, feature extraction and segmentation calculations are performed on the input synthetic image data to generate the location information, size and shape of the potential lesion area; Based on the potential lesion information obtained by model reasoning, a potential lesion detection result is generated; the potential lesion detection result includes lesion spatial coordinates, morphological parameters, and staging information.

[0127] In a specific implementation, the disease monitoring model can be a deep learning structure (such as UNet, Mask R-CNN, YOLO series, 3D ResNet, etc.), or a hybrid model that combines traditional image processing and machine learning algorithms.

[0128] It is understandable that, usually, this model has been trained or fine-tuned on a real case set or a data set containing real and synthetic images, and stored in a local or server environment in the form of a file (such as .pth, .h5, etc.). The system first loads the model in the inference phase to facilitate subsequent recognition or detection of synthetic image data.

[0129] Before the synthetic image is input into the model, several basic preprocessing operations need to be performed to ensure the correctness and consistency of the model input format. Exemplary preprocessing methods may include: resolution or size adjustment, normalization, channel number or dimension conversion, and cropping or center alignment.

[0130] Furthermore, during the inference phase, the disease monitoring model will perform convolution on the input image layer by layer (CNN scenario) or use structures such as visual Transformer to extract feature maps.

[0131] For example, if the model is of target detection type (such as YOLO, Faster R-CNN), the bounding box (Bounding Box) and confidence of the potential lesion area are output; If the model is of image segmentation type (such as UNet, Mask R-CNN), a pixel-level segmentation mask (Mask) of the lesion is generated, and the location, size and shape characteristics of the lesion can be further calculated.

[0132] Post-process the detection or segmentation output to extract key information such as the center coordinates, major and minor diameters, and contour morphology of the lesion in the image coordinate system. If a 3D model is used, volume and three-dimensional morphological features can also be obtained.

[0133] Through the above reasoning process, the location information (such as coordinates (x, y) or (x, y, z)), size (such as diameter, volume) and shape characteristics (such as number of lobes, burr conditions) of potential lesions in the synthetic images can be identified. This part of information can not only help verify the consistency of synthetic lesions with prior requirements, but also be compared with the results of real clinical cases to further evaluate the performance of the model.

[0134] After the inference and post-processing are completed, the various indicators are sorted and output to form a readable potential lesion detection result. For example, the result may include: lesion spatial coordinates: indicating the location of the lesion center point or segmentation mask; morphological parameters: such as long diameter, short diameter, contour boundary details, convexity, etc.; staging information: if the model contains staging discrimination logic or is associated with the medical knowledge graph, the clinical stage (such as stage I, stage II) that the lesion may be in can be output; confidence or score: the model scores the probability or consistency of the test result to assist users in evaluating its accuracy.

[0135] In this way, after the synthetic image data is successfully input into the preset disease monitoring model, the system can successfully complete the three stages of model loading and preprocessing, model reasoning, and result output. Finally, the potential lesion detection results are generated, including key information such as the lesion's spatial coordinates, morphological parameters, and staging information, to help users or clinicians quickly locate and evaluate the lesion's morphological characteristics and possible clinical stages.

[0136] The lesion synthesis and monitoring process of the present invention can not only make up for the pain point of insufficient real cases, but also play an important role in AI model evaluation or teaching; If the test results do not match the expected staging, they can also be fed back to the medical knowledge graph or generator to further strengthen the adversarial training or graph update mechanism and form a closed-loop "data-model-graph" mechanism.

[0137] As an optional implementation, in response to the staging information, lesion size, and morphological characteristics of the potential lesion detection result not being consistent with the multi-level prior information, the inconsistent information is recorded as updated data; The updated data is sent to the update interface of the medical knowledge graph to adjust the information of the target disease in the medical knowledge graph.

[0138] In the aforementioned implementation, it has been explained how to input synthetic image data into a preset disease monitoring model and obtain potential lesion detection results. In order to achieve continuous updating and dynamic improvement of the medical knowledge graph, the present invention further introduces a feedback mechanism of "difference information recording - medical knowledge graph update - generator / monitoring model parameter adjustment", so that when the detection results are inconsistent with the graph prior, correction or expansion can be performed automatically or semi-automatically.

[0139] In the specific implementation, after the model inference is completed, the system has obtained the potential lesion detection results such as lesion spatial coordinates, morphological parameters and staging information (see the above implementation method for details). The detection results are matched and compared with the corresponding prior information stored in the medical knowledge graph (such as lesion size, morphological coding, staging threshold, etc.). If the detection results are significantly different from the medical knowledge graph in size, morphology, and staging, they are marked as "inconsistent" or "new knowledge point".

[0140] If the system detects this inconsistency or finds potential new morphological features, it immediately triggers the entry point of the medical knowledge graph update process and packages all relevant data for the next step of recording and analysis; This test comparison can be performed in automatic mode or reviewed by technicians or clinicians in semi-automatic mode to avoid writing noise or obvious abnormal errors back into the medical knowledge graph.

[0141] Furthermore, when comparing the potential lesion detection results with the prior information, if it is found that the “stage is beyond the specified range” or the “morphological features are not in the originally defined interval”, the difference information is recorded in a structured form, such as “deviation amount”, “newly added morphological description”, “inferred stage”, etc.; For example, for stage III lung cancer, the diameter is a priori required to be ≥25 mm, but the detected diameter is only 22 mm. The difference item can be recorded as "the diameter is less than 3 mm" and the potential stage can be "inferred to be stage III but the size does not meet the standard."

[0142] The difference information is saved in the system database or cache, and the unique detection result ID and the node ID (or relationship ID) of the prior information are associated. This ensures that the data is traceable and facilitates subsequent review or expert audit.

[0143] If it is confirmed that the test results show "lesion boundary morphology not recorded in the priori" or "larger / smaller staging interval", it can be supplemented or modified in the attributes of the corresponding nodes (such as "morphological characteristics of stage III lung cancer") or relationships (such as "morphological stage association") in the atlas; For example, if multiple tests show that the diameter of a stage III tumor can be as low as 22 mm, but the atlas originally specified it as ≥25 mm, the lower limit of the stage size can be modified to 22 mm, or a new annotation "the lower limit can be extended to 22 mm" can be added to the attributes.

[0144] If a morphology that has never been recorded before appears, such as a rare honeycomb edge in the advanced stage of lung cancer, the system can add a "honeycomb morphology" node to the medical knowledge graph and establish an association with the "lung cancer stage IV" node to record its possibility and conditions; In this way, the medical knowledge graph can continuously absorb new clinical features or abnormal staging manifestations, so that subsequent lesion synthesis and detection have richer prior support.

[0145] In a specific implementation, the original version of the prior information can be retained when performing an update, and the new version can be marked as "extended" or "revised", so that clinicians or researchers can trace the change process and its reasons.

[0146] When the medical knowledge graph is updated, the generator will load new prior restriction rules or temporal attribute intervals in the next round of training or fine-tuning to correct the morphological distribution of lesion image synthesis; If a new staging threshold is added to the medical knowledge graph, then in the training loss function, the penalty for lesion synthesis cases that exceed or fall below the threshold will be strengthened or the restrictions will be relaxed.

[0147] If the newly discovered staging or morphological features appear with a certain frequency in the real environment, the disease monitoring model (such as deep learning detection / segmentation network) can also be fine-tuned with a small amount of incremental training on new synthetic images or real data; This will enhance the ability to identify rare stages or new morphologies, allowing the model to continuously evolve and adapt to the latest clinical knowledge.

[0148] After multiple rounds of testing, it was found that a certain staging threshold was too small, which caused a lot of "difference information" and the medical knowledge graph was revised after review. The next round of training of the generator will use an updated threshold to synthesize more lesion images that conform to the actual clinical situation. After retraining the disease monitoring model using these synthetic images, the recognition rate of the stage was significantly improved. The whole process forms a closed-loop iteration of "medical knowledge graph-generator-monitoring model".

[0149] In this way, the present invention can quickly adapt and gradually improve when faced with multiple stages, rare lesions or emerging new clinical discoveries. Its ultimate goal is to build an intelligent ecosystem that can interact and learn with the medical knowledge graph in both synthesis and detection.

[0150] In summary, this application deeply integrates the medical knowledge graph with the generative adversarial network (GAN), and through the comprehensive use of multi-level prior information such as lesion size, morphology, temporal progression law, and stage restrictions, it significantly enhances the simulation and detection capabilities of multi-stage and multi-morphological lesions. By determining the lesion stage restriction rules and morphological dependencies, the generator more accurately generates lesion images that meet clinical priors during adversarial training, reducing the appearance of invalid or unreasonable lesion morphologies, and improving the acceptability of synthetic images in clinical scenarios. The generator is iteratively optimized by fusing the difference assessment loss and the compliance judgment loss with a preset weight. Compared with the traditional single GAN training method, it can better ensure the synchronous approximation of lesions in spatial morphology and stage characteristics, and provide higher quality training and reasoning data for subsequent disease monitoring models. In the case of inconsistent monitoring results or new morphologies, the system can feed back to the medical knowledge graph for updating or fine-tuning when necessary, so as to iteratively improve the adaptability to rare lesions or complex stage characteristics, greatly enhancing the system's self-learning and expansion capabilities. Through multi-algorithm fusion and pixel-level alignment, the lesion images that meet the multi-level prior information requirements are smoothly fused with the original medical images, and then input into the disease monitoring model, so as to achieve better detection and segmentation effects even in scenarios with rare lesions or insufficient scarce staging data, thus meeting the diversified clinical needs.

[0151] Therefore, this application takes into account both morphological constraints and clinical interpretability in the integrated generation-detection process by organically combining lesion synthesis, staging priors, and monitoring models, improves the accuracy, applicability, and scalability of existing technologies, and overall enhances the efficiency and reliability of medical image recognition and auxiliary diagnosis.

[0152] Based on the same inventive concept, the embodiments of the present disclosure also provide a disease monitoring system based on medical image data identification corresponding to the disease monitoring method based on medical image data identification. Since the principle of problem solving by the system in the embodiments of the present disclosure is similar to the above-mentioned disease monitoring method based on medical image data identification in the embodiments of the present disclosure, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be repeated.

[0153] Reference Figure 3 , which is a schematic diagram of a disease monitoring system based on medical image data recognition provided by an embodiment of the present disclosure, the system includes: an acquisition module 10, a first processing module 20, a second processing module 30, and a generation module 40; The acquisition module 10 is used to obtain multi-level prior information related to the target disease from the medical knowledge graph, and the multi-level prior information includes: Lesion attributes, including lesion size, shape, location, and anatomical structure dependency; At least one temporal attribute describing the progression of the target disease; and lesion boundary and staging restriction rules for restricting the lesion morphology and staging of the target disease; The first processing module 20 is used to use a generative adversarial network as the lesion generation model, and the generative adversarial network includes a generator and a discriminator; Encoding the lesion attributes and time series attributes in the multi-level prior information into a knowledge prior vector and inputting it into the generator, so that the generator outputs lesion image data based on the knowledge prior vector to match the anatomical structure and progression law of the target disease; While the discriminator performs a difference evaluation between the lesion image data output by the generator and the knowledge prior vector, the discriminator also calls the lesion boundary and staging restriction rules in the medical knowledge graph to perform compliance judgment on the morphology and staging status of the lesion image data; The second processing module 30 is used to update the network parameters of the generator through multiple rounds of adversarial training based on the results of the difference evaluation and compliance determination output by the discriminator until the lesion image data that meets the multi-level prior information requirements is obtained; The generating module 40 is used to combine the lesion image data satisfying the multi-level prior information requirements with the original medical image, and input the data into a preset disease monitoring model to generate a potential lesion detection result.

[0154] Those skilled in the art can understand that in the above method of the specific implementation, the order of writing each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic. It should be understood that determining B based on A does not mean determining B based only on A, but B can also be determined based on A and / or other information.

[0155] In the description of this specification, the description with reference to the terms "exemplary", "for example", "particularly", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

Claims

1. A disease monitoring method based on medical image data recognition, characterized in that: The method comprises: Obtain multi-level prior information related to the target disease from the medical knowledge graph, the multi-level prior information includes: Lesion attributes, including lesion size, shape, location, and anatomical structure dependency; At least one temporal attribute describing the progression of the target disease; and lesion boundary and staging restriction rules for restricting the lesion morphology and staging of the target disease; Using a generative adversarial network as the lesion generation model, the generative adversarial network includes a generator and a discriminator; Encoding the lesion attributes and time series attributes in the multi-level prior information into a knowledge prior vector and inputting it into the generator, so that the generator outputs lesion image data based on the knowledge prior vector to match the anatomical structure and progression law of the target disease; While the discriminator performs a difference evaluation between the lesion image data output by the generator and the knowledge prior vector, the discriminator also calls the lesion boundary and staging restriction rules in the medical knowledge graph to perform compliance judgment on the morphology and staging status of the lesion image data; Based on the results of the difference evaluation and compliance determination output by the discriminator, the network parameters of the generator are updated through multiple rounds of adversarial training until lesion image data that meets the multi-level prior information requirements is obtained; The lesion image data that meets the multi-level prior information requirements is combined with the original medical image and input into a preset disease monitoring model to generate potential lesion detection results.

2. The disease monitoring method based on medical image data recognition according to claim 1, characterized in that: The step of causing the generator to output lesion image data based on the knowledge prior vector comprises: Numerically processing the size, shape, position and anatomical structure dependency of the lesion attributes to generate a lesion attribute vector, wherein the lesion attribute vector is used to characterize the spatial characteristics and anatomical priors of the target disease lesion; Performing time series feature extraction on the time series attributes describing the progression law of the target disease, and fusing the obtained time series features with the lesion attribute vector to form a time series fusion vector, wherein the time series fusion vector is used to characterize the dynamic changes of the lesion at different stages and progression stages; In the time series fusion vector, based on the critical threshold and stage interval defined by the lesion boundary and the staging restriction rule obtained from the medical knowledge graph, a priori restriction processing is performed to mark and restrict the time series fusion vector in the lesion morphology and staging dimensions; The temporal fusion vector that has been processed with prior constraints is input into the generator, and the lesion spatial features and progression stage information contained in the temporal fusion vector are used to generate lesion image data during the training and reasoning stages.

3. The disease monitoring method based on medical image data recognition according to claim 2, characterized in that: The step of performing a difference evaluation between the lesion image data output by the generator and the knowledge prior vector comprises: Extracting lesion image features from the lesion image data output by the generator through a feature extraction algorithm to form an image feature vector; Compare the image feature vector with the knowledge prior vector one by one in the spatial attribute dimension, the temporal progression dimension and the stage restriction dimension, and calculate the difference measure respectively; A difference evaluation result is determined according to the difference measurement results of each dimension, and the difference evaluation result is used to indicate the deviation of the lesion image data from the knowledge prior vector.

4. The disease monitoring method based on medical image data recognition according to claim 3 is characterized in that: The compliance determination of the morphology and staging status of the lesion image data includes: Based on the boundary restriction rules obtained from the medical knowledge graph, the edge contour in the lesion image data is matched with a preset lesion morphology threshold, and a morphology non-compliance indication is output when the lesion morphology exceeds the lesion morphology threshold; Based on the staging interval in the time series attribute, the staging status of the lesion image data is compared with the staging restrictions in the medical knowledge graph, and when the staging status exceeds the staging interval, a staging non-compliance indication is output; The results of morphological compliance and stage compliance determination are integrated to form a compliance determination result, and the compliance determination result is used to guide the continuation or correction of the adversarial training process.

5. The disease monitoring method based on medical image data recognition according to claim 4, characterized in that: The updating of the network parameters of the generator through multiple rounds of adversarial training based on the difference evaluation and compliance determination results output by the discriminator includes: Obtaining a difference loss based on the difference assessment result, obtaining a compliance loss based on the compliance determination result, and fusing the difference loss and the compliance loss with a preset weight to generate a total loss; During the training process, the generator is optimized by gradient back-propagation; When the total loss is lower than a preset threshold or the number of training rounds reaches an upper limit, the updating of the network parameters of the generator is stopped to obtain lesion image data that meets the requirements of the multi-level prior information.

6. The disease monitoring method based on medical image data recognition according to claim 5, characterized in that: The combining of the lesion image data satisfying the multi-level prior information requirements with the original medical image comprises: Based on the position attributes in the lesion image data, the lesion pixel coordinates are converted and aligned to the corresponding coordinate positions of the original medical image; After the coordinates are aligned, a preset pixel fusion process is performed on the boundary area where the lesion area overlaps with the original medical image, wherein the pixel fusion process includes performing a gradient correction or a seamless cloning operation on the fusion boundary; After completing the pixel fusion processing of the boundary area, the merged image data is output as the synthetic image data.

7. The disease monitoring method based on medical image data recognition according to claim 6, characterized in that: The step of combining the lesion image data that meets the multi-level prior information requirements with the original medical image and inputting the data into a preset disease monitoring model to generate potential lesion detection results includes: Loading a preset disease monitoring model and performing image preprocessing operations on the synthetic image data; In the model inference stage, feature extraction and segmentation calculations are performed on the input synthetic image data to generate the location information, size and shape of the potential lesion area; Based on the potential lesion information obtained by model reasoning, a potential lesion detection result is generated; the potential lesion detection result includes lesion spatial coordinates, morphological parameters, and staging information.

8. The disease monitoring method based on medical image data recognition according to claim 7, characterized in that: Also includes: In response to the staging information, lesion size, and morphological characteristics of the potential lesion detection result not being consistent with the multi-level prior information, recording the inconsistent information as updated data; The updated data is sent to the update interface of the medical knowledge graph to adjust the information of the target disease in the medical knowledge graph.

9. A disease monitoring system based on medical image data recognition, which is used to implement the disease monitoring method based on medical image data recognition according to any one of claims 1 to 8, characterized in that: include: A collection module, a first processing module, a second processing module, and a generation module; The acquisition module is used to obtain multi-level prior information related to the target disease from the medical knowledge graph, and the multi-level prior information includes: Lesion attributes, including lesion size, shape, location, and anatomical structure dependency; At least one temporal attribute describing the progression of the target disease; and lesion boundary and staging restriction rules for restricting the lesion morphology and staging of the target disease; The first processing module is used to use a generative adversarial network as the lesion generation model, and the generative adversarial network includes a generator and a discriminator; Encoding the lesion attributes and time series attributes in the multi-level prior information into a knowledge prior vector and inputting it into the generator, so that the generator outputs lesion image data based on the knowledge prior vector to match the anatomical structure and progression law of the target disease; While the discriminator performs a difference evaluation between the lesion image data output by the generator and the knowledge prior vector, the discriminator also calls the lesion boundary and staging restriction rules in the medical knowledge graph to perform compliance judgment on the morphology and staging status of the lesion image data; The second processing module is used to update the network parameters of the generator through multiple rounds of adversarial training based on the results of the difference evaluation and compliance determination output by the discriminator until the lesion image data that meets the multi-level prior information requirements is obtained; The generation module is used to combine the lesion image data that meets the multi-level prior information requirements with the original medical image, and input it into a preset disease monitoring model to generate potential lesion detection results.

Images