A big data-based medical auxiliary AI management method
By managing samples through a two-way flow mechanism and a three-level buffer pool, the problem of insufficient sample value assessment in traditional methods is solved, achieving efficient resource utilization and improved model performance, especially in improving diagnostic accuracy for marginal cases and difficult samples.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2026-04-21
- Publication Date
- 2026-06-05
Smart Images
Figure CN122158010A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data technology, and in particular to a medical AI management method based on big data. Background Technology
[0002] In the medical field, with the rapid development of medical imaging technology, the amount of medical imaging data generated by imaging equipment such as X-rays, CT scans, and MRIs has exploded. This imaging data contains a wealth of disease information and plays a crucial role in the early detection, diagnosis, and treatment planning of diseases. However, faced with massive amounts of medical imaging data, traditional manual image interpretation methods are not only inefficient but also easily affected by factors such as the doctor's experience and fatigue, making it difficult to guarantee the accuracy and consistency of diagnostic results.
[0003] In the training process of medical imaging AI models, the accuracy and consistency of labeled data have a crucial impact on model performance. However, due to the complexity of medical images and the diversity of disease manifestations, errors and inconsistencies are inevitable during the labeling process. These labeling errors directly affect the model's training effectiveness, leading to misdiagnosis and missed diagnosis in practical applications. Therefore, effectively managing and optimizing training data to improve the accuracy and consistency of labeled data has become a key issue in enhancing the performance of medical imaging AI models.
[0004] Traditional data management methods often indiscriminately store all manually verified and deemed valuable rejected samples in a single buffer pool and reintroduce them into the training process with a uniform and extremely low learning rate. The drawback of this approach is the lack of fine-grained differentiation and dynamic evaluation of the potential value of samples within the buffer pool, relying solely on a single metric (such as time) or static threshold for management. This leads to low recovery efficiency for high-value samples, uneconomical resource allocation, and a lack of ongoing utility evaluation of reintroduced samples. Once a sample enters the reintroduction process, it cannot be downgraded or eliminated based on its subsequent actual contribution. This easily results in a large accumulation of invalid or low-marginal-benefit samples in the buffer pool, consuming valuable memory and computing resources and interfering with efficient model optimization. Furthermore, traditional methods often completely discard rejected samples, failing to fully utilize the information they contain. In reality, rejected samples often contain information about repeated model errors on similar problems. This information is crucial for identifying the model's weaknesses and improving its accuracy in identifying marginal and difficult cases. Therefore, how to systematically mine and utilize the information in the filtered samples has become an important direction for further improving the performance of medical imaging AI models. Summary of the Invention
[0005] This application provides a big data-based medical auxiliary AI management method. Through a two-way flow mechanism, high-value samples are rapidly promoted and reused, while samples with diminishing value are promptly downgraded, avoiding resource waste. This enables intelligent, two-way flow and full lifecycle management of training samples, forming an adaptive, self-purifying, and resource-efficient data management closed loop. This improves the stability and efficiency of the model optimization process and achieves fully automated intelligent management.
[0006] This application provides a big data-based AI-assisted medical management method, including: S101: Collect medical image annotation information and training sample information, calculate the accuracy based on the collected medical image annotation information and training sample information, and identify samples to be screened based on the accuracy. S102, construct a three-level annular buffer pool and observation area based on the sample to be screened. The three-level annular buffer pool includes a cold buffer pool, a warm buffer pool and a hot buffer pool. According to the forward rule in the set flow rules, the sample to be screened is flowed between the three-level annular buffer pools to form a forward flow path. S103: Collect the loss value of the samples in the hot buffer pool in real time, determine whether negative flow is triggered based on the loss value, remove the samples that trigger the conditions from the hot buffer pool and put them into the observation area, and use the negative rules of the flow rules to downgrade the samples and form a negative flow path. S104, after the samples to be screened undergo forward and reverse flow, the determined samples to be screened are obtained. The screened samples are removed from the sample data, and medical image AI annotation is performed based on the sample data after removal.
[0007] Preferably, the method for identifying samples to be screened based on accuracy is as follows: statistically analyze the judgment results of all experts on each label, calculate the proportion of experts who judge each label as accurate to the total number of experts, and use this as the accuracy of the label. Set a screening threshold for the label accuracy according to the requirements of training the medical imaging AI model. Iterate through all labeled samples, compare the accuracy of each sample label with the screening threshold, and mark samples with an accuracy lower than the threshold as samples to be screened.
[0008] Preferably, the positive rule in the flow rule refers to comparing the confidence level of samples in the cold buffer pool with the first threshold by setting a first threshold and a second threshold. Samples with a confidence level greater than the first threshold are promoted to the warm buffer pool, and samples in the warm buffer pool are compared with the second threshold. Samples with a confidence level greater than the second threshold are promoted to the hot buffer pool.
[0009] Preferably, the steps for forming a positive flow path are as follows: when a sample that has been promoted to the hot buffer pool through the positive rules in the flow rules accumulates over time and meets the training cycle conditions, a sample is selected from the hot buffer pool and moved into the observation area. The selected sample will be introduced into the main training set. In the observation area, the sample is used for actual model training and testing. If, during the observation period, the model uses the selected sample at a rate greater than or equal to a preset accuracy threshold, the selected sample is introduced into the main training set; otherwise, the selected sample is moved back from the observation area to the hot buffer pool. The flow process from the cold buffer pool to the warm buffer pool, from the warm buffer pool to the hot buffer pool, and from the hot buffer pool to the observation area constitutes the positive flow path.
[0010] Preferably, the method for determining whether to trigger negative flow based on the loss value is as follows: set a loss value threshold, compare the loss value of the real-time monitored sample with the predicted loss value threshold, if the loss value of the sample is less than the predicted loss value threshold, proceed to variance judgment, otherwise, do not proceed to variance judgment; by calculating the variance of the sample loss value sequence, set a variance threshold, when the obtained variance is less than the preset variance threshold, determine that negative flow is triggered.
[0011] Preferably, the management method further includes: S201, obtain the time and silence duration of the sample entering the hot buffer pool, and calculate the silence index based on the time and silence duration of the sample entering the hot buffer pool; S202, the hot buffer pool is divided into three regions according to the quiescent index, and flow rules are set based on the quiescent index. The sample flows in the three regions according to the flow rules; the three regions are the active region, the standby region and the quiescent region. S203, for samples in the silent zone, a downgrade process is triggered according to the set elimination rules. The samples are downgraded using the downgrade method in step S103, the downgraded data is screened out, and medical image AI annotation is performed based on the removed sample data.
[0012] Preferably, the elimination rule is set as follows: the repartitioning period is set to R and the number of consecutive stay periods is M. The repartitioning period stipulates that the hot buffer pool samples are repartitioned once after each R rounds of training. The number of consecutive stay periods is set so that samples that have stayed in the silent zone for more than M repartitioning periods will be judged as long-term silent invalid samples in the hot pool. At the end of each repartitioning period, all samples in the silent zone are sorted out and the number of repartitioning periods that each sample has stayed in the silent zone is recorded. By comparing this number with the preset value of M, if the number of repartitioning periods that a sample has stayed in the silent zone is greater than or equal to the preset value of M, the sample is judged to meet the elimination condition; otherwise, it does not meet the elimination condition.
[0013] Preferably, the management method further includes: S301, Obtain the screened samples from steps S104 and S203, perform error classification based on the metadata of the screened samples, and construct an error trajectory sample library based on the classification results; S302, cluster the error trajectory sample library to generate weak regions, classify the generated weak regions, identify active weak regions by error severity, extract error samples and challenge samples based on active weak regions, assign weights to error samples and challenge samples, and form training batches with error samples, challenge samples and corresponding weights. S303, Insert reinforcement training into regular training. The reinforcement training refers to training the weak areas using training batches. After reinforcement training, use the reinforced model to adjust the error trajectory sample library, re-identify the weak areas, adjust the risk level of the weak areas, and adjust the strategy of the AI annotation model.
[0014] Preferably, the method for extracting error samples and challenge samples based on active weak regions is as follows: for each active weak region, a distance metric is used to select several samples that are closest to the cluster center as error samples. By calculating the loss value of the model on each sample, several samples in the cluster that have made the most serious mistakes are selected as challenge samples. In terms of quantity, the number of several is twice the number of multiple.
[0015] One or more technical solutions provided in this application have at least the following technical effects or advantages: through a bidirectional flow mechanism, high-value samples are rapidly promoted and reused, while samples with diminishing value are promptly downgraded, avoiding resource waste, realizing intelligent, bidirectional flow and full lifecycle management of training samples, forming an adaptive, self-purifying, and resource-efficient data management closed loop, improving the stability and efficiency of the model optimization process, and realizing fully automated intelligent management. The elimination rules can automatically and promptly clean up long-term silent invalid samples in the hot pool, keep the hot buffer pool capacity light, accelerate data turnover, and enhance the self-optimization capability at the micro-scheduling level. This allows inefficient samples to be identified and preprocessed within the hot pool, greatly alleviating the pressure on the main elimination process and improving the response speed and cleaning efficiency of the entire system. This enables the optimal allocation of training resources and real-time recycling of invalid occupancy within the hot pool. By systematically mining erroneous samples and conducting targeted reinforcement training, the model's recognition accuracy on long-tailed distributions, marginal cases, and difficult samples can be effectively improved. The discarded samples and their logs generated during the training data management process are transformed into localizable and describable weak areas of the model. An adaptive reinforcement training mechanism is designed to actively repair these weak areas, systematically improving the model's annotation accuracy on marginal cases, difficult cases, and rare types, and reducing the risk of clinical implementation. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating a big data-based AI-assisted medical management method according to the present invention. Figure 2 This is a schematic diagram of the process for calculating the silence index in this invention; Figure 3 This is a schematic diagram of the process for identifying weak areas in this invention. Detailed Implementation
[0017] To facilitate understanding of the present invention, a more complete description of this application will be given below with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to enable a more thorough and complete understanding of the disclosure of the present invention.
[0018] It should be noted that the terms "vertical," "horizontal," "up," "down," "left," "right," and similar expressions used in this article are for illustrative purposes only and do not represent the only possible implementation.
[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.
[0020] Example 1: Existing technical solutions indiscriminately store all manually verified and valuable rejected samples in a single buffer pool and reintroduce them into training at a uniform and extremely low learning rate. This method has two core flaws: it fails to finely distinguish and dynamically evaluate the potential value of samples in the buffer pool, relying solely on a single indicator (such as time) or static threshold for management. This results in low recovery efficiency of high-value samples and uneconomical resource allocation. The system is designed as a one-way screening-reintroduction process, lacking continuous utility evaluation of reintroduced samples. Once a sample enters the reintroduction process, it cannot be downgraded or eliminated based on its subsequent actual contribution, easily leading to a large accumulation of invalid or extremely low marginal benefit samples in the buffer pool, consuming valuable memory and computing resources, and interfering with efficient model optimization.
[0021] Figure 1 This is a flowchart illustrating a big data-based AI-assisted medical management method according to an embodiment of the present invention, including: S101: Collect medical image annotation information and training sample information, calculate the accuracy based on the collected medical image annotation information and training sample information, and identify samples to be screened based on the accuracy. Specifically, the process involves acquiring labeled medical image data from a medical image annotation system. Through interface integration with this system, image files are downloaded in batches according to a specific data request protocol. Simultaneously, corresponding annotation information is obtained, including lesion type and size, annotation personnel information, and annotation time. Annotator information includes the annotator's name, professional background, and work experience, while the annotation time records the exact moment the annotation was completed. Sample information for training the medical image AI model is collected, with samples from different hospitals and departments. Parameters of the image acquisition equipment are recorded, including the equipment model, imaging mode, and scanning parameters. The number of collected training samples is statistically analyzed to clarify the association between each training sample and its corresponding annotation information. A unique identifier is established to link sample files and annotation files, ensuring data integrity and consistency. An evaluation standard is established using the AI chip (LX MAX), and medical experts are organized to work with the LX MAX-based system. Experienced annotators, having accumulated expertise through multiple pre-annotation operations in simulated annotation scenarios, jointly established accurate evaluation standards for medical image annotations. A certain percentage (10%) of the collected annotated medical image data was randomly sampled to form an evaluation sample set. Medical experts, utilizing the LX MAX chip, reviewed the annotations in the evaluation sample set and, according to the established evaluation standards, determined the accuracy (accurate or inaccurate) of each annotation. The results of all expert judgments for each annotation were statistically analyzed, and the proportion of experts who judged each annotation as accurate out of the total number of experts was calculated as the accuracy rate of that annotation. Based on the requirements of medical image AI model training and actual application scenarios, a screening threshold for annotation accuracy was set. All annotated samples were iterated through, and the accuracy rate of each sample was compared with the screening threshold. Samples with an accuracy rate lower than the threshold were marked as samples to be screened.
[0022] S102, construct a three-level annular buffer pool and observation area based on the sample to be screened. The three-level annular buffer pool includes a cold buffer pool, a warm buffer pool and a hot buffer pool. According to the forward rule in the set flow rules, the sample to be screened is flowed between the three-level annular buffer pools to form a forward flow path. Furthermore, based on the collected training sample information, a main training set is determined. The determination of the main training set comprehensively considers the model training objectives, ensuring it covers the basic data range and feature distribution required for model training. After determining the main training set, a three-level annular buffer pool and observation area are constructed according to the samples to be screened. The three levels of annular buffer pools are a cold buffer pool, a warm buffer pool, and a hot buffer pool. Arranged from left to right according to the direction of sample data flow, the cold buffer pool, warm buffer pool, and hot buffer pool form a closed annular structure. The cold buffer pool, as the starting point of the annular structure, has the largest capacity and can be imagined as a wide annular band, capable of accommodating a large number of initially screened samples with potential retention value. The warm buffer pool, located after the cold buffer pool, has a medium capacity and is an annular band of medium width, used to store samples that have been promoted from the cold buffer pool and have high model prediction confidence. The hot buffer pool, as the ending point of the annular structure, has the smallest capacity and is an annular band of narrow shape, used to store high-value samples that will participate in model training. The samples to be screened are classified and managed through the three-level annular buffer pool and observation area. The cold buffer pool serves as the initial collection... The regions for samples to be screened include the cold buffer pool, where samples are in an evaluation state. These samples, due to initial screening criteria, are not directly used for training but may have potential value. During initialization, all initial samples to be screened are stored in the cold buffer pool. The warm buffer pool is reserved for samples that have passed initial evaluation and show potential value. These samples, after a certain evaluation, exhibit characteristics that positively impact model training but have not yet met the criteria for direct inclusion in the main training set. The warm buffer pool serves as a transitional space and is empty during initialization. The hot buffer pool stores samples that have undergone rigorous evaluation and are confirmed to have high value and are about to be reintroduced into training. These samples, after multiple rounds of evaluation, have had their characteristics and value fully verified and have high potential to improve model performance. The hot buffer pool is empty during initialization. The observation area is a temporary region used to store samples removed from the hot buffer pool that are undergoing final utility verification. Before samples are introduced into the main training set, they are observed and evaluated in the observation area to ensure that their actual effect on model training meets expectations and to avoid affecting model performance due to the introduction of unsuitable samples. The observation area is empty during initialization. The positive rule in the transfer rules refers to comparing the confidence level of samples in the cold buffer pool with the first threshold by setting a first threshold and a second threshold. Samples with a confidence level greater than the first threshold are promoted to the warm buffer pool. Similarly, the confidence level of samples in the warm buffer pool is compared with the second threshold, and samples with a confidence level greater than the second threshold are promoted to the hot buffer pool. The specific conversion process is as follows: a certain number of samples are drawn from the cold buffer pool according to a fixed training cycle frequency. The model is used to evaluate the sampled samples and calculate the predicted confidence level of the samples. The sampled samples are input into the model, which will sequentially pass through convolutional layers, pooling layers, and fully connected layers to calculate and transform the input data, and finally obtain the model's predicted confidence level of the sample. The predicted confidence level reflects the model's certainty about the category to which the sample belongs. A first threshold is set, and the predicted confidence level of the sample is compared with the preset first threshold. If the predicted confidence level of the sample exceeds the first threshold, it indicates that the sample has certain potential value, and the sample is promoted from the cold buffer pool to the warm buffer pool. This process is repeated every 3... The training cycle is used as an interval to evaluate samples in the warm buffer pool. The evaluation cycle of the warm buffer pool is longer than that of the cold buffer pool. The confidence performance of the samples in N consecutive evaluations (e.g., 2 times) is checked. A second threshold is set, which is greater than the first threshold. If the confidence of a sample exceeds the second threshold in N consecutive evaluations, it is considered to have high value and is promoted to the hot buffer pool. When the samples in the hot buffer pool have accumulated for a certain period of time and meet the training cycle conditions, a portion of the samples are selected from the hot buffer pool and moved to the observation area. These samples are introduced into the main training set. In the observation area, the samples are used for actual model training and testing to observe their impact on model performance. A series of evaluation metrics, such as accuracy, are used to measure the performance. The changes in model performance before and after the introduction of a sample are analyzed. If, during the observation period, the model uses a sample with an accuracy greater than or equal to the preset accuracy threshold, then the sample has practical value and can be formally introduced into the main training set. Conversely, if the sample has a negative impact on model performance or the improvement effect is not significant, then the sample is moved back from the observation area to the hot buffer pool or other processing is performed according to the specific situation. The above-mentioned flow process from the cold buffer pool to the warm buffer pool, from the warm buffer pool to the hot buffer pool, and from the hot buffer pool to the observation area constitutes a positive flow path.
[0023] S103: Real-time collection of loss values of samples in the hot buffer pool; triggering negative flow based on loss values; removing samples that meet the triggering conditions from the hot buffer pool and placing them in the observation area; using the negative rules of the flow rules to downgrade the samples and form a negative flow path. Specifically, during each model training process, the loss value of each sample in the hot buffer is recorded in real time. The loss values from multiple training iterations of each sample are then used to form a loss value sequence. This loss value is calculated by inputting the model's predicted output and the sample's true label (the true value of the sample's target variable) into a pre-defined loss function. Based on the real-time monitored loss value of each sample in the hot buffer, it is determined whether a negative flow is triggered. Specifically, a loss value threshold is set, and the real-time monitored loss value of the sample is compared with the predicted loss value threshold. If the sample's loss value is less than the predicted threshold... The loss value threshold is set, and if it exceeds the threshold, the variance is evaluated; otherwise, it is not evaluated. The variance of the sample loss value sequence is calculated, and a variance threshold is set. When the obtained variance is less than the preset variance threshold, it is determined that the model has fully grasped the knowledge of that sample, thus triggering negative flow. The negative flow process is as follows: the sample triggering negative flow is removed from the hot buffer and placed in a pre-defined observation area. The transfer time and training history information of the sample in the hot buffer are recorded. The sample in the observation area undergoes several training observation periods. Experimental and control groups are set up to verify the samples in the observation area. To demonstrate this, the experimental group continued training the model using the normal procedure excluding the observation sample. That is, in subsequent training iterations, samples were selected from the hot buffer (which no longer contained the observation sample) and other relevant data sources for training, keeping other training parameters and conditions unchanged. The control group's experimental environment was exactly the same as the experimental group. After the observation period, the observation sample was added back into the training process in the normal manner. This means that during training, in addition to the samples used by the experimental group, the observation sample was also added. After the observation period ended and the experimental and control groups completed their tests, the model performance of the two groups was compared. The comparison metrics can be selected according to the specific task and model characteristics, such as accuracy, recall, F1 score, mean squared error, etc. Based on the comparison results, the following decisions are made: If the model performance of the experimental group is not lower than that of the control group, it indicates that the sample has no significant contribution to the model training. At this time, the sample is downgraded. The downgrade process is the opposite of the positive flow, that is, the hot buffer pool sample is transferred to the warm buffer pool, and the warm buffer pool sample is transferred to the cold buffer pool to form a negative flow path. If the model performance of the experimental group is lower than that of the control group, it indicates that the sample still has key value. No downgrade is performed, and the sample is returned to the hot buffer pool.
[0024] S104, after the samples to be screened undergo forward and reverse flow, the determined samples to be screened are obtained. The screened samples are removed from the sample data, and medical image AI annotation is performed based on the sample data after removal.
[0025] Furthermore, after dual screening through positive and negative flow, samples that were not included in the main training set were identified as screening samples. These screening samples were removed from the sample data. The remaining sample data after removing the screening samples has high quality and representativeness, and can better reflect the characteristics and distribution of medical images. According to the professional requirements and standards of medical image AI annotation, these high-quality sample data are accurately annotated. Annotators accurately mark and describe lesions, tissue structures, etc. in the images according to pre-set annotation rules and standards. These annotated data will be used as training samples for the model. Steps S101 to S104 are executed to obtain the sample data after removing the screening samples. The sample data after removing the screening samples is used to help the model learn the characteristics and patterns of medical images, thereby improving the diagnostic accuracy and reliability of the model.
[0026] The technical solutions in the above embodiments of this application have at least the following technical effects or advantages: through the bidirectional flow mechanism, high-value samples are quickly promoted and reused, and samples with depreciating value are promptly downgraded, avoiding resource waste, realizing intelligent, bidirectional flow and full life cycle management of training samples, forming an adaptive, self-purifying, and resource-efficient data management closed loop, improving the stability and efficiency of the model optimization process, and realizing fully automated intelligent management.
[0027] Example 2: Building upon the bidirectional sample flow in Example 1, an indiscriminate, continuous reintroduction strategy is employed. This results in a large number of silent samples remaining unused in the hot pool for extended periods, yet still occupying cache resources. There is a lack of proactive and forward-looking identification and cleanup mechanisms for this type of inactive, ineffective occupation. This example addresses this by introducing state awareness and frequency modulation capabilities into the hot buffer pool, enabling gradient resource scheduling where high-activity samples are trained at high frequencies, while low-activity samples are maintained at low frequencies or prepared for withdrawal. Figure 2 As shown.
[0028] S201, obtain the time and silence duration of the sample entering the hot buffer pool, and calculate the silence index based on the time and silence duration of the sample entering the hot buffer pool; Furthermore, when a sample is promoted from the warm buffer or other data sources to the hot buffer according to specific flow rules, the time of entry into the hot buffer is recorded using a Unix timestamp. The moment the sample enters the hot buffer is stored in the sample's metadata, which is a set of data describing the sample's relevant attributes and features. At the instant a sample enters the hot buffer, its most recent activity time (i.e., the moment it entered the hot buffer) is obtained. Whenever a sample is sampled for model training, its most recent activity time is immediately updated, as participation in training is a sign of activity. Recording the time of each activity accurately reflects the sample's usage in the hot buffer. The most recent activity time is updated to the current system time. The quiet duration measures the time elapsed since the sample was last sampled for training. A longer quiet duration indicates a longer period of inactivity, potentially resulting in a smaller contribution to model training; conversely, a shorter quiet duration indicates a more active sample, possibly with higher value. The formula for calculating the quiet duration is: ,in, The duration of silence. The current system time is the current moment when the silence duration is calculated. The time when a sample was last sampled for training is recorded, along with the time of its last active period. The time of entry into the hot buffer and the duration of silence are normalized. The silence index is calculated using these two factors: the time of entry into the hot buffer and the duration of silence. The formula is as follows: + (1- ),in, The quiescence index is used to comprehensively measure the duration of a sample's silence and its recent usage benefits in order to determine the sample's activity and value. The weighting coefficient for the duration of silence. These are the weighting coefficients for the benefit assessment values, and Greater than , + =1, The duration of silence. The benefit evaluation value records the average change in the performance index of the model in the most relevant category to the sample during the subsequent validation after the sample has participated in training N times. If the sample has participated in training less than N times, the average value is calculated based on the actual number of times. When the calculated quiescent index approaches zero, it indicates that the sample has been used recently or has significant benefits in recent use, belonging to a high-activity and high-value state. When the quiescent index approaches 1, it indicates that the sample has been dormant for a long time and has "mediocre benefits in recent use, belonging to a low-activity and low-efficiency state."
[0029] S202, the hot buffer pool is divided into three regions according to the quiescent index, and flow rules are set based on the quiescent index. The sample flows in the three regions according to the flow rules; the three regions are the active region, the standby region and the quiescent region. Specifically, the hot buffer pool is divided into three regions based on the quiescent index: an active region, a standby region, and a quiescent region. The active region contains the top one-third of the samples based on the quiescent index. This one-third refers to calculating the quiescent index for each sample in the hot buffer pool, arranging the calculated quiescent indices in ascending order, and assigning all samples in the top one-third to the active region. Samples in the active region have the highest activity level and must participate in every training round. In this way, the model can continuously learn from these active and high-value samples, improving training effectiveness and model performance. The standby region contains the middle one-third of the samples based on the quiescent index. Samples in the standby region have a moderate level of activity, neither the most active nor the longest idle time. Samples in the standby region participate using a probability-based approach. A base participation probability is set for the standby region. Before each training round, a random number is generated for each sample in this region. Based on the comparison between the random number and the base participation probability, it is determined whether the sample will participate in the current round of training. A sample whose random number of training iterations is greater than the base participation probability will participate in the current round; otherwise, it will not participate. The silent zone contains the last one-third of the samples with the lowest silent index. Samples in the silent zone have the longest idle time and the lowest activity level. The silent zone implements a low-frequency participation strategy, and samples in the silent zone do not participate in training. Only when the model's performance on the validation set continuously declines will a small number of samples be temporarily extracted from this zone for wake-up evaluation. The method for setting flow rules based on the silent index is as follows: Set an active high threshold. When the silent index of a sample in the active zone is greater than the preset active high threshold, it indicates that the activity level of the sample has decreased and it no longer meets the characteristic requirements of the active zone sample. At this time, it will flow into the waiting zone. Set a waiting low threshold and a waiting high threshold. When the silent index value of a sample in the waiting zone is less than the waiting low threshold, it indicates that its activity level has increased and it will flow into the active zone. When the silent index value of the waiting zone is greater than the waiting high threshold, it indicates that its activity level has further decreased and it will flow into the silent zone. Samples in the silent zone that improve the model performance by more than half can be directly returned to the active zone; if the improvement is one-third, they will be returned to the waiting zone.
[0030] S203, for samples in the silent zone, a downgrade process is triggered according to the set elimination rules. The samples are downgraded using the downgrade method in step S103, the downgraded data is screened out, and medical image AI annotation is performed based on the removed sample data.
[0031] Furthermore, the elimination rules are specifically set as follows: The repartitioning period is set to R, and the number of consecutive stay periods is M. The repartitioning period stipulates that the hot buffer pool samples are repartitioned once after each R rounds of training. The number of consecutive stay periods is set so that samples that have stayed in the silent zone for more than M repartitioning periods are judged as long-term silent invalid samples in the hot buffer pool. At the end of each repartitioning period, all samples in the silent zone are reviewed, and the number of repartitioning periods each sample has stayed in the silent zone is recorded. This number is compared with a preset value of M. If the number of repartitioning periods a sample has stayed in the silent zone is greater than or equal to the preset value of M, the sample is judged to meet the elimination conditions; otherwise, it does not meet the elimination conditions. Samples that meet the elimination conditions are downgraded, and long-term silent invalid samples (i.e., samples that meet the elimination conditions) are directly removed from the hot buffer pool. The samples are sent as high-priority candidates to the negative flow in step S103. The samples are downgraded using the downgrade method in step S103. After the samples are downgraded to the cold buffer, they are evaluated and manually reviewed by professional medical personnel. Based on the quality evaluation results, samples that do not meet the quality requirements are screened out. Annotation tasks are assigned based on the data of the removed samples. The samples can be classified according to factors such as disease type and imaging modality, and then assigned to the corresponding annotators or annotation teams to ensure that the amount of annotation tasks undertaken by each annotator or team is appropriate and the annotation difficulty matches their professional capabilities.
[0032] The technical solutions in the above embodiments of this application have at least the following technical effects or advantages: the elimination rules can automatically and timely clean up long-term silent invalid samples in the hot pool, keep the hot buffer pool capacity light, accelerate data turnover, enhance the self-optimization capability at the micro-scheduling level, and enable inefficient samples to be identified and preprocessed inside the hot pool, which greatly alleviates the pressure of the main elimination process, improves the response speed and cleaning efficiency of the entire system, and thus realizes the optimal allocation of training resources and real-time recycling of invalid occupancy inside the hot pool.
[0033] Example 3: Examples 1 and 2 completely discarded the screened samples, causing the model to repeatedly make mistakes on similar problems. The decision boundary remained unclear in weak areas, thus limiting the theoretical upper limit of the accuracy of medical image AI annotation. This example treats the discarded samples as high-value diagnostic signals, rather than simply negative outputs. Figure 3 As shown.
[0034] S301, Obtain the screened samples from steps S104 and S203, perform error classification based on the metadata of the screened samples, and construct an error trajectory sample library based on the classification results; Furthermore, a data transmission channel is established by interfacing with the sample screening data interface of Embodiments 1 and 2. Through this interface, screened samples are redirected to avoid being discarded according to the original process. The screening samples are then acquired via the transmission channel, and their metadata is collected. This metadata includes sample images and annotations, depth feature vectors, elimination reason codes, historical evaluation records, and final error prediction results. The sample images and annotations are located and retrieved from the original image database based on the sample's unique identifier information, while simultaneously extracting associated doctor annotation information. The depth feature vectors are extracted using a model that performs feature extraction on the sample images. The model, through algorithms and structures, transforms the sample images into depth feature vectors and saves the extracted depth feature vectors. The elimination reason codes record the specific reasons for sample elimination in detail, identified in code form. Common elimination reasons include silent timeout. The process involves persistently low confidence levels and manual review for elimination. By recording reason codes, the background and basis for sample rejection can be clearly understood. The historical evaluation record is obtained from the model's evaluation log, chronologically retrieving the most recent 10 confidence levels of the sample during the warm / hot pool period. The final error prediction result records the model's final prediction category and corresponding confidence level for the sample. Error classification is performed through metadata, specifically: a series of error type labels are defined to accurately describe different error situations that occur during the model prediction process. Common error type labels include boundary swing error, high confidence error, and rare error. The boundary swing error occurs in the classification boundary region, where the model's prediction results are unstable and may be classified into different categories in different evaluation rounds, reflecting the model's insufficient judgment ability at the classification boundary. The high confidence error refers to a high confidence level in the model's prediction of the sample, but the actual prediction result is incorrect. This indicates that the model makes incorrect judgments due to overconfidence in certain situations; the rare type refers to samples belonging to rare diseases or having rare characteristics. Because the model encounters fewer such samples during training, its ability to identify them is limited, making it prone to incorrect predictions. Based on the captured sample metadata features, the model uses preset rules and algorithms to make incorrect classifications.
[0035] Set the capacity of the error trajectory sample library to 10% of the active training set. Construct the error trajectory sample library based on classification results and metadata. Use a first-in-first-out (FIFO) strategy to update the error trajectory sample library. When the library capacity reaches the set limit, the system will automatically remove the oldest records in the library first to make room for newly added filtered samples.
[0036] S302, cluster the error trajectory sample library to generate weak areas, classify the generated weak areas, extract error samples and challenge samples based on the weak areas, assign weights to error samples and challenge samples, and form training batches with error samples, challenge samples and corresponding weights. Specifically, the trigger condition is set to complete 30 rounds of regular training. After every 30 rounds of training, a cluster analysis of the error trajectory sample library is triggered using the DBSCAN algorithm. The neighborhood radius and minimum sample size of the DBSCAN algorithm are set. After clustering, for clusters that pass the validity test (sample size ≥ 5), common features of samples within the clusters are extracted from two dimensions: image features and disease type features. Based on the feature analysis results, a description of weak areas is generated. For example, cluster C1: features include small lung nodules (<5mm) with a halo sign, and the main error type is false negative. The weak areas are generated through cluster analysis from... The weak regions, identified in the error trajectory sample library as sample sets with poor model performance and specific error patterns, serve to provide clear directions for model optimization. Targeted reinforcement training can improve model performance. Based on the average error confidence, error consistency, and potential impact on clinical diagnosis, weak regions are categorized into Level 1, Level 2, and Level 3. Level 1 weak regions, i.e., high-risk weak regions, exhibit high certainty in model error judgments within clusters, with frequent and clearly defined errors, highly similar sample error types, and severe concentration of model deficiencies in specific aspects. Level 2 weak regions, i.e., medium-risk weak regions, show some certainty in model error judgments within clusters. Obvious error tendencies and patterns; sample error types are relatively similar, model problems are relatively dispersed, for level three weak regions (i.e., focus on weak regions), the model's judgment of errors in samples within a cluster is highly uncertain, errors are sporadic or complex; sample error types have low similarity, model problems are not prominently concentrated, each weak region's key attribute information is recorded in detail, including the cluster center feature vector, which can represent the core features of the entire weak region, the number of samples, reflecting the size of the weak region, classification information, and clarifying the danger level of the weak region; the specific method for identifying active weak regions is: in terms of error severity, when samples in the weak region generally have high loss values and A sample with a large proportion of high-loss error values is considered active. In terms of development trend, if the error situation worsens with time, data volume, or changes in business scenarios, or if it has poor adaptability to new data and the error rate increases significantly, it belongs to an active weak area. For each active weak area, a distance metric is used to select the 10 samples closest to the cluster center as error samples. By calculating the model's loss value on each sample, the 5 samples with the most severe model errors (highest loss) in the cluster are selected as challenge samples. A small number of samples with similar characteristics to the active weak area but which the model can handle correctly are retrieved from the main training set as control samples.
[0037] The selected error samples and challenge samples are assigned reinforcement weights, which are determined based on the level of their respective weak areas and their own representativeness. For example, the weight of error samples in the first-level weak area is 2.0, and the weight of error samples in the second-level weak area is 1.5. For extreme challenge cases, 0.5 is added to the weight of the core samples. The reinforcement weight of the challenge samples in the first-level weak area is 2.0 + 0.5 = 2.5, and the reinforcement weight of the challenge samples in the second-level weak area is 1.5 + 0.5 = 2.0. All the samples selected in this round and their corresponding reinforcement weights are integrated to form training batches. The training batches are used to perform targeted reinforcement training on the model to improve the model's performance in the weak areas.
[0038] S303, Insert reinforcement training into regular training. The reinforcement training refers to training the weak areas using training batches. After reinforcement training, use the reinforced model to adjust the error trajectory sample library, re-identify the weak areas, adjust the risk level of the weak areas, and adjust the strategy of the AI annotation model.
[0039] Furthermore, in the conventional training process of training the AI model using the removed sample data in Embodiments 1 and 2 above, an insertion rule is set, specifically: after every 10 rounds of conventional training, one round of reinforcement training is inserted, and after the reinforcement training is completed, conventional training continues. For reinforcement training, a mixture of 20% targeted reinforcement samples and 80% conventional sampling samples is used. The targeted reinforcement samples are obtained from the training batch. During the reinforcement training process, the loss function and learning rate are adjusted. Focal Loss is used to calculate the loss for the targeted reinforcement samples, so that the model focuses more on samples in the weak areas during training. The learning rate is reduced, so that the model can adjust the decision boundary in a more refined and stable way, avoiding oscillations in the training process due to an excessively large learning rate, which would affect the improvement of model performance.
[0040] The enhanced model is used to adjust the error trajectory sample library, re-identify weak areas, adjust the risk level of weak areas, and after completing the enhancement training, the enhanced model is used to re-evaluate the samples belonging to each weak area in the error trajectory sample library using clustering methods. The change in the average error rate of each weak area sample is calculated, and the evaluation results are recorded in detail. The average error rate is one of the important indicators for measuring the performance of the model in the weak area. By comparing the change of this indicator before and after enhancement training, the effect of enhancement training can be intuitively understood. Based on the calculated change in the average error rate of each weak area sample, a judgment standard is set. Specifically, if the average error rate of a certain weak area is consistently below 5%, and this low error rate state is maintained for 3 cycles, then the state of the weak area is marked as regressed and removed from the active enhancement list. This indicates that after enhancement training, the performance of the model in the weak area has been significantly improved, and no further focus or enhancement training is needed. Based on the changing trend of the average error rate of each weak area sample, the risk level of the weak area is dynamically adjusted. For example, if the average error rate of a certain weak area continues to rise, it is upgraded from level 2 (medium risk) to level 1 (high risk).
[0041] Based on the feedback from the effectiveness of each weak area after reinforcement training, the parameters during the training process are dynamically adjusted. For example, if the training effect of a certain weak area is found to be poor, it is believed that the mixing ratio of targeted reinforcement samples is unreasonable. By appropriately increasing the mixing ratio of targeted reinforcement samples in the training batch for that weak area, the learning intensity of the model for that weak area sample is improved. The strategy of the AI annotation model is adjusted based on the performance of the model and the changes in the weak area after reinforcement training.
[0042] The technical solutions in the above embodiments of this application have at least the following technical effects or advantages: By systematically mining erroneous samples and conducting targeted reinforcement training, the model's recognition accuracy on long-tailed distributions, marginal cases, and difficult samples can be effectively improved. The eliminated samples and their logs generated during the training data management process are transformed into localizable and describable weak areas of the model. An adaptive reinforcement training mechanism is designed to actively repair these weak areas, systematically improve the model's annotation accuracy on marginal cases, difficult cases, and rare types, and reduce the risk of clinical implementation.
[0043] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A medical AI-assisted management method based on big data, characterized in that, include: S101: Collect medical image annotation information and training sample information, calculate the accuracy based on the collected medical image annotation information and training sample information, and identify samples to be screened based on the accuracy. S102, construct a three-level annular buffer pool and observation area based on the sample to be screened. The three-level annular buffer pool includes a cold buffer pool, a warm buffer pool and a hot buffer pool. According to the forward rule in the set flow rules, the sample to be screened is flowed between the three-level annular buffer pools to form a forward flow path. S103: Collect the loss value of the samples in the hot buffer pool in real time, determine whether negative flow is triggered based on the loss value, remove the samples that trigger the conditions from the hot buffer pool and put them into the observation area, and use the negative rules of the flow rules to downgrade the samples and form a negative flow path. S104, after the samples to be screened undergo bidirectional circulation, the determined samples to be screened are obtained. The screened samples are removed from the sample data, and medical image AI annotation is performed based on the sample data after removal. The bidirectional circulation includes positive circulation and negative circulation.
2. The medical AI management method based on big data according to claim 1, characterized in that, The step of identifying samples to be screened based on accuracy includes the following sub-steps: statistically analyzing the judgment results of all experts for each annotation, calculating the proportion of experts who judged each annotation as accurate to the total number of experts, using this as the accuracy of the annotation, setting a screening threshold for the annotation accuracy according to the requirements of training the medical imaging AI model, traversing all annotated samples, comparing the accuracy of each sample annotation with the screening threshold, and marking samples with an accuracy lower than the threshold as samples to be screened.
3. The medical AI management method based on big data according to claim 2, characterized in that, The positive rule in the flow rules refers to comparing the confidence level of samples in the cold buffer pool with the first threshold by setting a first threshold and a second threshold. Samples with a confidence level greater than the first threshold are promoted to the warm buffer pool. The confidence level of samples in the warm buffer pool is compared with the second threshold. Samples with a confidence level greater than the second threshold are promoted to the hot buffer pool.
4. The medical AI management method based on big data according to claim 3, characterized in that, The steps to form a positive flow path are as follows: When a sample that has been promoted to the hot buffer pool through the positive rules in the flow rules accumulates over time and meets the training cycle conditions, a sample is selected from the hot buffer pool and moved into the observation area. The selected sample will be introduced into the main training set. In the observation area, the sample is used for actual model training and testing. If, during the observation period, the model uses the selected sample with an accuracy greater than or equal to the preset accuracy threshold, the selected sample will be introduced into the main training set. Conversely, the selected sample is moved from the observation area back to the hot buffer pool. The process of transferring the sample from the cold buffer pool to the warm buffer pool, from the warm buffer pool to the hot buffer pool, and from the hot buffer pool back to the observation area constitutes the forward flow path.
5. The medical AI management method based on big data according to claim 4, characterized in that, The process of determining whether to trigger negative flow based on the loss value includes the following sub-steps: setting a loss value threshold; comparing the loss value of the real-time monitored sample with the predicted loss value threshold; if the loss value of the sample is less than the predicted loss value threshold, proceeding to variance judgment; otherwise, not proceeding to variance judgment; calculating the variance of the sample loss value sequence and setting a variance threshold; when the obtained variance is less than the preset variance threshold, determining that negative flow is triggered.
6. The medical AI management method based on big data according to claim 1, characterized in that, It also includes the following steps: S201, obtain the time and silence duration of the sample entering the hot buffer pool, and calculate the silence index based on the time and silence duration of the sample entering the hot buffer pool; S202, the hot buffer pool is divided into three regions according to the quiescent index, and flow rules are set based on the quiescent index. The sample flows in the three regions according to the flow rules; the three regions are the active region, the standby region and the quiescent region. S203, for samples in the silent zone, a downgrade process is triggered according to the set elimination rules. The samples are downgraded using the downgrade method in step S103, the downgraded data is screened out, and medical image AI annotation is performed based on the removed sample data.
7. The medical AI management method based on big data according to claim 6, characterized in that, The formula for calculating the silence index is: + (1- ),in, The silence index is used to comprehensively measure the duration of silence and recent usage benefits of a sample. The weighting coefficient for the duration of silence. These are the weighting coefficients for the benefit assessment values, and Greater than , + =1, The duration of silence. This is the benefit assessment value.
8. The medical AI management method based on big data according to claim 7, characterized in that, The elimination rules are set as follows: The repartitioning period is set to R, and the number of consecutive stay periods is set to M. The repartitioning period stipulates that the hot buffer pool samples are repartitioned once after each R rounds of training. The number of consecutive stay periods is set so that samples that have stayed in the silent zone for more than M repartitioning periods are judged as long-term silent invalid samples in the hot pool. At the end of each repartitioning period, all samples in the silent zone are reviewed, and the number of repartitioning periods each sample has stayed in the silent zone is recorded. This number is compared with a preset value of M. If the number of repartitioning periods a sample has stayed in the silent zone is greater than or equal to the preset value of M, the sample is judged to meet the elimination conditions; otherwise, it does not meet the elimination conditions.
9. The medical AI management method based on big data according to claim 6, characterized in that, It also includes the following steps: S301, Obtain the screened samples from steps S104 and S203, perform error classification based on the metadata of the screened samples, and construct an error trajectory sample library based on the classification results; S302, cluster the error trajectory sample library to generate weak regions, classify the generated weak regions, identify active weak regions by error severity, extract error samples and challenge samples based on active weak regions, assign weights to error samples and challenge samples, and form training batches with error samples, challenge samples and corresponding weights. S303, Insert reinforcement training into regular training, wherein the reinforcement training refers to training weak areas using training batches; After enhanced training, the enhanced model is used to adjust the error trajectory sample library, re-identify weak areas, adjust the risk level of weak areas, and adjust the strategy of the AI annotation model.
10. A medical AI management method based on big data according to claim 9, characterized in that, Extracting error samples and challenge samples based on active weak regions includes the following sub-steps: For each active weak region, using a distance metric, select several samples closest to the cluster center as error samples. By calculating the model's loss value on each sample, select several samples in the cluster where the model makes the most serious errors as challenge samples. In terms of quantity, the number of "several" samples is twice the number of "multiple" samples.