A multi-center data standardization method for evaluating AI ultrasound model performance

Abstract

The application provides a multi-center data standardization method for evaluating AI ultrasonic model performance, relates to the technical field of ultrasonic data processing, and comprises the following steps: collecting original ultrasonic image data from different data sources, and performing standardization preprocessing to obtain standardization preprocessing images; a multi-task quality evaluation model is constructed and pre-trained; the pre-trained multi-task quality evaluation model is applied to quantitatively evaluate the images to obtain quality evaluation indexes; based on the quality evaluation indexes, a batch-by-batch adaptive threshold strategy and a multi-level screening mechanism are adopted to perform quality screening to obtain qualified labeling images; and the qualified labeling images are sampled in layers to obtain a standardized multi-center ultrasonic evaluation dataset. The application can solve the problems of quality evaluation difficulty caused by heterogeneity of multi-center ultrasonic data sources, inability of a fixed threshold strategy to adapt to quality baseline differences of different data sources, lack of spatial position consideration in artifact evaluation, and absence of a multi-center data balanced sampling mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ultrasound data processing, and particularly relates to a multi-center data standardization method for evaluating AI ultrasound model performance. BACKGROUND

[0002] With the rapid development of artificial intelligence technology in the field of medical imaging, AI ultrasound large models have shown great potential in tasks such as auxiliary diagnosis and lesion detection. However, the performance evaluation of AI models is highly dependent on the quality and representativeness of the evaluation dataset. Existing ultrasound datasets are mostly derived from a single medical center or specific equipment, and it is difficult to reflect the data heterogeneity in real clinical scenarios. When the model is deployed to an external medical center, it often faces problems such as equipment differences and uneven image quality, resulting in a significant decline in performance. Therefore, it is an urgent problem to construct a standardized evaluation dataset covering multiple centers, multiple devices, and multiple diseases.

[0003] The ultrasound image data of external centers often has multiple quality problems. Differences in operation specifications, equipment maintenance levels, and technician experience among different medical centers result in a large number of incomplete, blurred, or severely artifacted samples in the images. Different brands and models of ultrasound equipment have significant differences in imaging principles, parameter settings, and image formats, and direct mixing will introduce systematic bias. Traditional quality screening uses fixed global thresholds to uniformly determine all batches of data, ignoring the quality baseline differences between different medical centers and equipment, resulting in excessive exclusion of low-quality batches of data and underutilization of high-quality batches of data. Existing artifact evaluation methods are too rough, simply using artifact area proportion as the determination standard, ignoring the influence of artifact location on diagnostic value. In addition, there is a lack of multi-center data balancing mechanism, and simply pursuing data quantity can easily lead to some data sources dominating the dataset while other sources being underrepresented, ultimately resulting in an evaluation set that cannot truly reflect the model's generalization ability in a multi-center scenario.

[0004] A subjective and objective ultrasound medical image quality evaluation method and system are disclosed in Chinese patent CN113628174A. The patent uses experts to perform subjective quality evaluation on ultrasound medical images and gives three classification results of low quality, medium quality and high quality, establishes a subjective evaluation database, then uses a deep neural network pre-trained on ImageNet to extract high-dimensional features of the images, sends the features to a fully connected layer to output three quality classification results, thereby constructing an automatic medical image quality evaluation system to assist doctors in screening unqualified images. The method proposed in this patent realizes automatic evaluation of ultrasound image quality based on deep learning, but this method only classifies the quality of images obtained by a single acquisition device and does not consider the influence of data source heterogeneity in a multi-center, multi-device scenario on quality evaluation, and lacks adaptive quality determination mechanisms for different data source batches and multi-center data balancing sampling strategies. SUMMARY

[0005] Therefore, the present application provides a multi-center data standardization method for evaluating the performance of AI ultrasound models to solve the problems of quality evaluation difficulty caused by multi-center ultrasound data source heterogeneity, fixed threshold strategy unable to adapt to different data source quality baseline differences, lack of spatial position consideration in artifact evaluation, and lack of multi-center data balancing sampling mechanism in the prior art, thereby constructing a standardized multi-center ultrasound evaluation data set with high quality, diversity and representativeness.

[0006] The technical solution of the present application is as follows:

[0007] The present application provides a multi-center data standardization method for evaluating the performance of AI ultrasound models, comprising:

[0008] S1, collecting original ultrasound image data from different data sources, performing desensitization processing and standardization preprocessing on the original ultrasound image data to obtain standardized preprocessing images;

[0009] S2, constructing and pre-training a multi-task quality evaluation model, the multi-task quality evaluation model comprising a target region detection module, an integrity scoring module, a clarity classification module and an artifact detection module;

[0010] S3, applying the pre-trained multi-task quality evaluation model to quantitatively evaluate all standardized preprocessing images to obtain quality evaluation indicators, and using a batch-by-batch adaptive threshold strategy and a multi-level screening mechanism based on the quality evaluation indicators to perform quality screening to obtain quality-qualified labeled images;

[0011] S4, stratified sampling the quality-qualified labeled images to obtain a standardized multi-center ultrasound evaluation data set.

[0012] Preferably, step S1 comprises:

[0013] Collecting raw ultrasound image data from multiple medical centers, different brands and models of ultrasound equipment, the raw ultrasound image data is stored in DICOM format;

[0014] Parsing each raw ultrasound image to extract image pixel data and imaging parameters, the imaging parameters including device brand, device model, probe type, imaging depth and gain parameter;

[0015] Using optical character recognition method to identify and remove sensitive information burned on the image, generating a unique anonymized identifier for each image and establishing an anonymized metadata record table retaining device model and imaging parameters;

[0016] Standardizing the desensitized image, scaling the original image of different sizes to a preset standard size while maintaining the aspect ratio, and filling the blank area generated by the aspect ratio adjustment with the background mean value;

[0017] Performing gray scale normalization processing, mapping the dynamic range of different devices to a standard gray space through linear mapping.

[0018] Preferably, the multi-task quality assessment model uses Vision Transformer as the backbone network, divides the input standardized preprocessed image into multiple image block sequences, and extracts global semantic features and local texture features through self-attention mechanism; a target region detection head is designed on the feature map output by the backbone network, and an anchor-based target detection method is used to output the target region detection box of the target organ or lesion; the integrity score module, the clarity classification module and the artifact detection module share global semantic features and local texture features for parallel processing.

[0019] Preferably, the integrity score module obtains the integrity score by calculating the intersection over union of the predicted target region detection box and the expert-annotated target region detection box; the clarity classification module outputs a binary classification label of clear or blurred based on image gradient features and texture features; the artifact detection module outputs a set of artifact region bounding boxes and corresponding artifact type labels, and the artifact types include motion artifact, gas artifact, reverberation artifact and acoustic shadow artifact; the multi-task quality assessment model is pre-trained using annotation data from multiple medical centers, including expert-annotated target region detection boxes, integrity scores, clarity labels, and artifact bounding boxes and types.

[0020] Preferably, the quality assessment indicators include integrity score, clarity classification label and spatially weighted artifact occlusion degree; wherein the spatially weighted artifact occlusion degree is a quantitative indicator obtained by partitioning the target region detection box and assigning different diagnostic importance weights.

[0021] Preferably, the partitioning of the target region bounding box and assigning different diagnostic importance weights include:

[0022] The target region bounding box is sequentially divided from the center to the core region, the transition region and the peripheral region. The core region is defined as a concentric rectangular region with the center as the reference and the area being 50% of the area of the target region bounding box. The transition region is defined as an annular region between the concentric rectangular region with the center as the reference and the area being 75% of the area of the target region bounding box and the core region. The peripheral region is defined as an annular region between the target region bounding box and the outer boundary box of the transition region. The diagnostic importance weights are assigned to the core region, the transition region and the peripheral region respectively , and , and .

[0023] Preferably, the calculation method of the artifact occlusion degree based on spatial weighting is:

[0024] For each detected artifact region bounding box, the intersection region with the target region bounding box is calculated. The number of pixels in the intersection region falling in the core region, the transition region and the peripheral region is counted. The weighted occlusion degree of a single artifact is calculated by weighted summation of the pixel number in each region and the corresponding diagnostic importance weight . When there are multiple artifacts, the intersection regions of all artifact region bounding boxes with the target region bounding box are merged to obtain the total occlusion region. The number of pixels in the total occlusion region falling in the core region, the transition region and the peripheral region is counted to calculate the cumulative weighted occlusion degree .

[0025] Preferably, the batch adaptive threshold strategy includes:

[0026] The standardized preprocessed images are grouped into several batches according to the data source, including medical center identifier and equipment model identifier;

[0027] The distribution characteristics of the quality evaluation indicators are calculated for each batch, including the central tendency statistics and dispersion degree statistics of the integrity score, the proportion of clear samples, the proportion of samples with artifacts, and the central tendency statistics and dispersion degree statistics of the cumulative weighted occlusion degree;

[0028] The global reference threshold is preset, including the global reference threshold of the integrity score and the global reference threshold of the cumulative weighted occlusion degree;

[0029] The batch quality deviation is calculated based on the difference between the distribution characteristics of each batch and the global reference threshold. The batch quality deviation includes the integrity deviation, the clarity deviation and the artifact occlusion deviation, which are used to quantify the deviation degree of the batch relative to the global quality baseline;

[0030] The batch quality deviation is calculated based on the difference between the distribution characteristics of each batch and the global benchmark threshold, and each batch is classified into a quality excellent batch, a quality medium batch and a quality low batch according to the batch quality deviation;

[0031] Differentiated determination thresholds are used for batches of different quality levels:

[0032] For a quality excellent batch, the integrity determination threshold of the batch is higher than the integrity score global benchmark threshold, and the occlusion degree determination threshold of the batch is lower than the cumulative weighted occlusion degree global benchmark threshold;

[0033] For a quality low batch, the integrity determination threshold of the batch is lower than the integrity score global benchmark threshold, and the occlusion degree determination threshold of the batch is higher than the cumulative weighted occlusion degree global benchmark threshold;

[0034] For a quality medium batch, the global benchmark threshold is used as the determination threshold of the batch.

[0035] Preferably, the multi-level screening mechanism includes: a first level of rapid pass screening, a second level of refined comprehensive evaluation and a third level of manual review and suggestion labeling;

[0036] In the first level of rapid pass screening stage, images that are superior to the batch determination threshold in the integrity score, the clarity classification label and the single artifact maximum weighted occlusion degree are determined as high-quality samples and directly retained;

[0037] In the second level of refined comprehensive evaluation stage, for images that do not pass the first level of screening, if any of the integrity score is lower than the batch determination threshold, the clarity classification label is fuzzy, or the cumulative weighted occlusion degree exceeds the batch determination threshold, it is determined as unqualified sample and excluded; for images that meet the batch determination threshold in the three quality dimensions, a comprehensive quality score is calculated, the comprehensive quality score is obtained by weighted fusion of the integrity exceeding degree, the clarity label and the occlusion degree redundancy degree, and the images are classified into high-quality samples, usable samples, samples to be audited or unqualified samples according to the value of the comprehensive quality score;

[0038] In the third level of manual review and suggestion labeling stage, review suggestion information is generated for images determined as samples to be audited, including the closeness of each quality dimension to the threshold, the specific position and type of artifact occlusion, the confidence of clarity evaluation and the quality dimension that the expert should focus on.

[0039] Preferably, in step S4, the step of stratified sampling of the labeled images that meet the quality includes:

[0040] Multi-dimensional quota constraints are established, including total sample number constraint, medical center coverage constraint, equipment brand coverage constraint and quality level distribution constraint;

[0041] According to the medical center source, equipment brand and type, and disease type, the qualified image is layered, and in each layer, the image with high quality level and high comprehensive quality score is selected in descending order according to the quality level and comprehensive quality score, and the image with high quality level and high comprehensive quality score is selected to fill the quota target number of the layer;

[0042] The target quota number is increased for the layer with high quality, and the target quota number is reduced for the layer with low quality, or the sample to be audited is required to be included in the data set after manual review;

[0043] In the process of selecting images to fill the quota layer by layer, if the number of high-quality samples in the layer is insufficient, the available samples and the samples to be audited are supplemented in turn, and the satisfaction of the overall quality level distribution constraint is monitored in real time, if the supplement leads to the unsatisfaction of the constraint, the supplement of the layer is stopped and the quota surplus is allocated to other layers;

[0044] Complete metadata labeling is added to each image finally included in the data set, and a statistical report of the data set is established.

[0045] The present application has the following beneficial effects compared with the prior art:

[0046] (1) The present application collects original ultrasound images from multiple centers and multiple equipment sources, and performs standardization preprocessing, constructs a multi-task quality evaluation model to realize multi-dimensional automatic quantitative evaluation of image quality, adopts a batch adaptive threshold strategy and a multi-level screening mechanism for quality control, and constructs a standardized multi-center ultrasound evaluation data set through a hierarchical sampling mechanism under multi-dimensional quota constraints, solves the problems of quality evaluation difficulty caused by multi-center data heterogeneity, fixed threshold strategy cannot adapt to different data source quality baseline differences, and multi-center data balance mechanism is missing, and the constructed evaluation data set can truly reflect the generalization ability of the AI ultrasound model in the multi-center clinical scene;

[0047] (2) The present application divides the target region from the center to the outer peripheral region into a core region, a transition region and an outer peripheral region, and gives different diagnosis importance weights according to the importance of different spatial positions to clinical diagnosis, calculates the artifact occlusion degree by weighted summation, so that the artifact in the core region has higher contribution to the occlusion degree than the artifact in the edge region, and accurately reflects the actual influence of artifact position on diagnosis value. Compared with the traditional simple area ratio determination method, the spatial weighted evaluation method can avoid misjudging large-area low-impact artifacts located at the edge as serious quality problems, effectively identify small artifacts that block key diagnosis regions, reduce data waste or misjudgment caused by rough evaluation, and improve the accuracy of image quality evaluation and data utilization;

[0048] (3) The application calculates batch quality deviation according to the difference between the distribution characteristics of each batch quality evaluation index and the global benchmark threshold, classifies the batches into quality excellent batches, quality medium batches and quality low batches, and uses differentiated judgment thresholds for batches of different quality levels, so that batches with higher quality baseline use higher judgment requirements and batches with lower quality baseline use relatively loose judgment standards. The adaptive threshold strategy can adapt to the quality baseline differences between different medical centers and equipment, avoid the problem of excessive exclusion of low-quality batch data and insufficient use of excellent quality batch data caused by traditional fixed global threshold, improve the coverage and representativeness of multi-center data set under the premise of ensuring controllable overall data quality;

[0049] (4) The application establishes a three-level progressive quality screening process, the first level directly retains high-quality samples with three quality dimensions significantly better than the judgment threshold through rapid screening, the second level performs fine comprehensive evaluation on the boundary samples by calculating the comprehensive quality score, and the third level generates review suggestion information containing the closeness of each quality dimension and the suggested focus for attention for the samples marked as samples to be audited. The multi-level screening mechanism can quickly identify and retain high-quality samples, finely and comprehensively judge the boundary samples to reduce misjudgment, and provide expert review basis for suspicious samples, improving screening efficiency while ensuring screening accuracy and data utilization;

[0050] (5) The application establishes a multi-dimensional quota system including total sample number constraint, medical center coverage constraint, equipment brand coverage constraint and quality level distribution constraint, stratifies the quality qualified images according to medical center source, equipment brand and disease type, preferentially selects high-quality samples in each layer, fills the quota with images with high comprehensive quality score, and dynamically adjusts the quota number in each layer according to the batch quality level. The stratified sampling mechanism strictly controls the number of samples from each source and the overall quality distribution while ensuring the diversity requirements of covering multiple centers, multiple equipment and multiple diseases, solves the contradiction between the lack of high-quality samples and the quality requirements by monitoring the overall quality level distribution constraint and dynamically adjusting the quota allocation, and ensures the representativeness and controllability of the evaluation data set. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0052] Figure 1 The method flowchart of the application;

[0053] Figure 2 Flow chart of data collection and desensitization processing of the present application;

[0054] Figure 3 Schematic diagram of three-region division of target region of the present application;

[0055] Figure 4 Schematic diagram of quality assessment flow of the present application;

[0056] Figure 5 Schematic diagram of batch screening and stratified sampling flow of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0058] As shown in Figure 1 The present application provides a multi-center data standardization method for evaluating the performance of an AI ultrasound model, comprising: S1, collecting original ultrasound image data from different data sources, performing desensitization processing and standardization preprocessing on the original ultrasound image data to obtain standardization preprocessed images; S2, constructing and pre-training a multi-task quality assessment model, the multi-task quality assessment model comprising a target region detection module, an integrity scoring module, a clarity classification module and an artifact detection module; S3, applying the pre-trained multi-task quality assessment model to quantitatively evaluate all standardization preprocessed images to obtain quality assessment indicators, and performing quality screening based on the quality assessment indicators using a batch-adaptive threshold strategy and a multi-level screening mechanism to obtain quality-qualified labeled images; S4, stratified sampling the quality-qualified labeled images to obtain a standardized multi-center ultrasound evaluation dataset.

[0059] As shown in Figure 2 In an embodiment of the present application, step S1 comprises:

[0060] Collecting original ultrasound image data from multiple medical centers, different brands and models of ultrasound equipment, the original ultrasound image data being stored in DICOM format;

[0061] Analyzing each original ultrasound image to extract image pixel data and imaging parameters, the imaging parameters including device brand, device model, probe type, imaging depth and gain parameter;

[0062] sensitive information including patient name, patient ID number, examination number, hospital name and doctor name are recognized and removed using optical character recognition technology;

[0063] A unique anonymization identifier is generated for each image, and an anonymization metadata record table is established to retain device model and imaging parameters, ensuring data traceability without involving patient privacy;

[0064] The desensitized image is standardized and preprocessed, and the original image of different sizes is uniformly scaled to a preset standard size while maintaining the aspect ratio;

[0065] In a specific embodiment, the preset standard size is 512 pixels by 512 pixels, and the blank area generated by aspect ratio adjustment is filled with background mean value;

[0066] Perform gray scale normalization processing to uniformly map the dynamic range of different devices to the standard gray space of 0 to 255 through linear mapping, and the linear mapping formula is:

[0067] ;

[0068] wherein is the original pixel value, and are the minimum and maximum pixel values of the image, respectively, is the normalized pixel value. The standardized preprocessed image is obtained through the above processing.

[0069] In an embodiment of the present application, step S2 comprises:

[0070] The multi-task quality assessment model uses Vision Transformer as the backbone network, divides the input standardized preprocessed image into multiple image block sequences, and extracts the global semantic features and local texture features of the image through the self-attention mechanism;

[0071] A target region detection head is designed on the feature map output by the backbone network, and an anchor-based target detection method is used to output the target region detection frame of the target organ or lesion, and the target region detection frame is expressed as a boundary box coordinate wherein is the upper left corner coordinate, is the lower right corner coordinate;

[0072] The integrity scoring module, the clarity classification module and the artifact detection module share the global semantic features and local texture features for parallel processing;

[0073] The integrity scoring module calculates the predicted target region detection frame and the expert-labeled target region detection frame The intersection-union ratio yields the integrity score. The formula for calculating the intersection-union ratio is:

[0074] ;

[0075] in Indicates the area of ​​the region. Indicates intersection, Represents the union;

[0076] The sharpness classification module outputs binary labels of sharpness or blurriness based on image gradient and texture features. These sharpness classification labels are denoted as follows: , where 0 represents blurry and 1 represents clear;

[0077] The artifact detection module outputs a set of bounding boxes for artifact regions and corresponding artifact type labels. The set of bounding boxes for artifact regions is denoted as... ,in The number of artifacts detected. For the first The bounding box coordinates of each artifact, and the artifact types include motion artifacts, gas artifacts, reverberation artifacts and sound artifacts;

[0078] The multi-task quality assessment model was pre-trained using labeled data from multiple medical centers. The labeled data included expert-annotated target region bounding boxes, integrity scores, sharpness labels, and artifact bounding boxes and types.

[0079] During training, a combined loss function is used, which consists of the mean squared error loss for integrity scoring, the binary cross-entropy loss for sharpness classification, and the multi-label cross-entropy loss for artifact detection. The loss function is expressed as follows:

[0080] ;

[0081] in The mean squared error loss for integrity scoring, For clarity classification, binary cross-entropy loss, For artifact detection, multi-label cross-entropy loss, , and These are weighting coefficients, which are set to 0.4, 0.3, and 0.3 respectively in a specific embodiment;

[0082] The Adam optimizer is used to update model parameters. In one specific embodiment, the initial learning rate is set to 0.001, and a cosine annealing strategy is used to dynamically adjust the learning rate. The training epochs are 100, and the batch size is 16. After training, a pre-trained multi-task quality assessment model is obtained.

[0083] In one embodiment of the present invention, step S3 includes:

[0084] The standardized pre-processed image is input into the pre-trained multi-task quality assessment model for inference to obtain the integrity score of each image , a clarity classification label , a target region bounding box , and a set of artifact region bounding boxes and corresponding artifact type labels.

[0085] For each image, if artifacts are detected, i.e. , the spatial weighting method is used to calculate the artifact occlusion degree:

[0086] As shown in Figure 3 , the target region bounding box is divided into a core region, a transition region, and a peripheral region from the center outward.

[0087] Let the center coordinates of the target region bounding box be , the width be , and the height be , then the core region is defined as a concentric rectangular region with an area of 50% of the area of the target region bounding box, with the center as the reference, and the specific boundary box coordinates are:

[0088] ;

[0089] where the coefficient ensures that the area of the core region is 50% of the area of the target region;

[0090] The transition region is defined as an annular region between the concentric rectangular region with an area of 75% of the area of the target region bounding box and the core region, with the center as the reference, and the specific outer boundary box coordinates are:

[0091] ;

[0092] where the coefficient ensures that the area of the outer boundary box is 75% of the area of the target region, and the transition region is the annular region between the outer boundary box and the core region boundary box;

[0093] The peripheral region is defined as the annular region between the target region bounding box and the outer boundary box of the transition region.

[0094] The diagnostic importance weights , , and , the diagnostic importance weight reflects the importance of different spatial regions to clinical diagnosis, the core region contains the key anatomical structure of the lesion or target organ with the highest weight, the transition region contains auxiliary diagnostic information with a medium weight, and the peripheral region mainly contains background or edge information with the lowest weight, satisfying In a specific embodiment, ;

[0095] As shown in Figure 4 , for each detected artifact region bounding box , calculate its intersection region with the target region detection box ;

[0096] Count the number of pixels of the intersection region falling in the core region, the transition region and the peripheral region respectively, denoted as , and ;

[0097] Calculate the total number of pixels of the core region, the transition region and the peripheral region in the target region detection box, denoted as , and ;

[0098] According to the weighted sum of the pixel number of each region and the corresponding diagnostic importance weight, the weighted occlusion degree of a single artifact is calculated , and the weighted occlusion degree calculation formula is:

[0099] ;

[0100] When there are multiple artifacts, the overlap between the artifacts needs to be considered. For each pixel in the target region, the contribution of being occluded is calculated only once to avoid repeated calculation. Specifically, first, the intersection regions of all artifact region bounding boxes and the target region detection box are merged to obtain the total occlusion region , then the number of pixels of the total occlusion region falling in the core region, the transition region and the peripheral region is counted, denoted as , and , the cumulative weighted occlusion degree is calculated, and the cumulative weighted occlusion degree calculation formula is:

[0101] ;

[0102] If no artifact is detected in the image, i.e. , the single artifact weighted occlusion degree and the cumulative weighted occlusion degree of the image are both set to 0;

[0103] After the above processing, each standardized pre-processed image obtains complete quality evaluation indicators, including integrity score , clarity classification label , single artifact maximum weighted occlusion degree , and cumulative weighted occlusion degree .

[0104] The above artifact occlusion degree calculation method based on spatial weighting realizes fine quantification of artifact position and occlusion degree by partitioning the target region according to diagnostic importance and assigning differentiated weights. Compared with the simple area ratio method, it can accurately reflect the actual impact of artifacts on clinical diagnostic value.

[0105] As shown in Figure 5 , the quality screening is performed based on the quality evaluation indicators using a batch-by-batch adaptive threshold strategy and a multi-level screening mechanism. The standardized pre-processed images are grouped into batches according to data sources, including medical center identifier and device model identifier, to ensure that images in the same batch come from the same medical center and the same type of ultrasound device;

[0106] The distribution characteristics of the quality evaluation indicators are calculated for each batch. The mean , standard deviation , 25th percentile , median , and 75th percentile of the integrity scores of all images in the batch are calculated;

[0107] The proportion of clear samples in the batch is calculated , which is the ratio of the number of samples with a clear sample proportion of to the total number of samples in the batch;

[0108] The proportion of samples with detected artifacts in the batch is calculated , which is the ratio of the number of samples with a detected artifact proportion of to the total number of samples in the batch;

[0109] For samples with detected artifacts, the mean , standard deviation , and median of the cumulative weighted occlusion degree are calculated.

[0110] Based on the batch distribution characteristics, an adaptive threshold strategy is used to determine the quality determination threshold for the batch:

[0111] A global reference threshold is preset, including an integrity score global reference threshold and a cumulative weighted occlusion degree global reference threshold. In a specific embodiment, the integrity score global reference threshold is 0.7, and the cumulative weighted occlusion degree global reference threshold is 0.2. , cumulative weighted obscuration global reference threshold ;

[0112] calculating batch quality deviation degree, the batch quality deviation degree including integrity deviation degree , clarity deviation degree and artifact obscuration deviation degree , wherein 0.70 is a global expected reference of the proportion of clear samples, used to quantify the deviation degree of the batch relative to the global quality baseline;

[0113] classifying each batch into a quality excellent batch, a quality medium batch and a quality low batch according to the batch quality deviation degree, and adopting differentiated determination thresholds for batches of different quality levels;

[0114] In a specific embodiment, if the integrity deviation degree and the clarity deviation degree and the artifact obscuration deviation degree , the batch is determined as a quality excellent batch, the integrity determination threshold of the batch is higher than the integrity score global reference threshold, and the obscuration degree determination threshold of the batch is lower than the cumulative weighted obscuration global reference threshold, specifically , , and the batch is given a higher quota weight in subsequent stratified sampling;

[0115] If the integrity deviation degree or the clarity deviation degree or the artifact obscuration deviation degree , the batch is determined as a quality low batch, the integrity determination threshold of the batch is lower than the integrity score global reference threshold, and the obscuration degree determination threshold of the batch is higher than the cumulative weighted obscuration global reference threshold, specifically , , and the quota weight of the batch is reduced or the proportion of manual review samples is increased in subsequent stratified sampling;

[0116] If the batch quality deviation degree is between the above two cases, the batch is determined as a quality medium batch, and the global reference threshold is adopted as the determination threshold of the batch, specifically and .

[0117] The above batch-adaptive threshold strategy dynamically adjusts the determination threshold according to the quality distribution characteristics of different medical centers and device batches, avoiding the problems of excessive exclusion of quality low batch data or insufficient utilization of quality excellent batch data caused by fixed global threshold, and improving the multi-center coverage and representativeness of the data set.

[0118] The multi-stage screening mechanism is adopted to determine the quality of all images in the batch, and the multi-stage screening mechanism includes a first-stage fast pass screening, a second-stage fine comprehensive evaluation, and a third-stage manual review and suggestion labeling.

[0119] In the first-stage fast pass screening stage, images that are significantly better than the batch determination threshold in the integrity score, the clarity classification label, and the single artifact maximum weighted occlusion degree are determined as high-quality samples and directly retained. In a specific embodiment, the fast pass condition is and and The design principle of the fast pass condition is that all three quality dimensions are significantly better than the batch determination threshold, ensuring that these samples have extremely high diagnostic value and the screening determination is uncontroversial.

[0120] In the second-stage fine comprehensive evaluation stage, comprehensive evaluation is performed on images that do not pass the first-stage screening. If any of the following conditions is not met, i.e., or or the image is determined as a substandard sample and excluded. For images that meet the batch determination threshold in all three quality dimensions, a comprehensive quality score is calculated as The comprehensive quality score is obtained by weighted fusion of the integrity excess degree, the clarity label, and the occlusion degree redundancy degree, and the comprehensive quality score calculation formula is:

[0121] ;

[0122] wherein , and are weight coefficients, which in a specific embodiment take values of 0.35, 0.30, and 0.35 respectively, ensuring that the contributions of the three quality dimensions to the comprehensive score are relatively balanced. The value range of the comprehensive quality score is 0 to 1, and the higher the value, the better the image quality.

[0123] According to the value of the comprehensive quality score , the images are classified into high-quality samples, usable samples, samples to be audited, or substandard samples;

[0124] In a specific embodiment, if , the image is determined as a high-quality sample and retained, and the labeling quality level is high quality; if , the image is determined as a usable sample and retained, and the labeling quality level is usable, with the addition of artifact position, artifact type, weighted occlusion degree value, and use suggestion, which includes a suggestion to be used only for robustness testing or to be used for training after labeling the artifact area; if If the sample is deemed unsatisfactory, it will be retained and marked with a quality level of "unsatisfactory," along with a boundary sample identifier and a suggestion for manual review; if If so, it is determined to be an unqualified sample and excluded;

[0125] In the third-level manual review and suggestion annotation stage, review suggestion information is generated for images identified as samples to be reviewed. This information includes the proximity of each quality dimension to a threshold, the specific location and type of artifact occlusion, the confidence level of the sharpness assessment, and the quality dimensions that experts should focus on. In one specific embodiment, for images that simultaneously meet the condition of a comprehensive quality score close to the lower bound of usable samples... Furthermore, this batch is of relatively low quality and requires as many qualified samples as possible to meet the quota requirements. It is marked as a high-priority review target.

[0126] The aforementioned multi-level screening mechanism achieves a progressive quality control process by quickly identifying high-quality samples through automated screening at the first level, making refined comprehensive judgments on boundary samples through refined comprehensive evaluation at the second level, and providing expert review suggestions for marking samples as questionable through manual review at the third level. This improves screening accuracy and data utilization.

[0127] After the above multi-level screening mechanism, a set of images that have passed the quality screening is obtained. Each image in the set is labeled with a quality level, a quality assessment index value, and usage suggestions or review suggestions. These images and their labeling information are recorded as labeled images that have passed the quality screening.

[0128] In one embodiment of the present invention, step S4 includes:

[0129] Establish multi-dimensional quota constraints, which include total sample size constraints, medical center coverage constraints, equipment brand coverage constraints, and quality level distribution constraints.

[0130] In one specific embodiment, the total sample size constraint requires the standardized multi-center ultrasound evaluation dataset to contain at least 5000 high-quality ultrasound images. The medical center coverage constraint requires the dataset to cover no fewer than 10 medical centers, with each center contributing between 300 and 800 images, avoiding an excessively high or low sample proportion from any single medical center. The equipment brand coverage constraint requires the dataset to cover no fewer than 5 mainstream ultrasound equipment brands, with each brand contributing at least 500 images. The quality level distribution constraint requires that the proportion of high-quality samples in the dataset be no less than 60%, the proportion of usable samples no more than 30%, and the proportion of samples awaiting review no more than 10%.

[0131] The quality qualified labeled images are nested and layered according to medical center sources, equipment brands and types, and disease types. In each layer, the images are sorted in descending order according to the quality grades and the comprehensive quality scores. The images with high quality grades and high comprehensive quality scores are preferentially selected to fill the quota target number of the layer. For the layer corresponding to the quality excellent batch determined in step S3, the target quota number of the layer is increased. For the layer corresponding to the quality low batch determined in step S3, the target quota number of the layer is reduced or kept but the samples to be audited are required to be included in the dataset after manual review;

[0132] In the process of selecting images to fill the quota by layer, if the number of high-quality samples in a layer is sufficient, only high-quality samples are selected to fill the quota of the layer. If the number of high-quality samples in a layer is insufficient, available samples and samples to be audited are supplemented in turn. In the supplement process, the satisfaction of the overall quality grade distribution constraint is monitored in real time. If the supplement leads to the unsatisfaction of the constraint, the supplement of the layer is stopped and the quota surplus is allocated to other layers, ensuring that the final dataset meets the multi-center diversity coverage requirement and meets the overall quality control standard;

[0133] Complete metadata labeling is attached to each image included in the final dataset, including anonymized identifier, medical center identifier, equipment brand and model, imaging parameters, disease type, quality grade, integrity score, clarity label, artifact location and type, single artifact maximum weighted occlusion degree, cumulative weighted occlusion degree, comprehensive quality score, and use suggestion or review suggestion;

[0134] A statistical report of the dataset is established, including the sample number distribution of each medical center, the sample number distribution of each equipment brand, the sample number distribution of each disease type, the sample number distribution of each quality grade, and the overall quality indicators of the dataset. The overall quality indicators include the average integrity score, the proportion of clear samples, the average cumulative weighted occlusion degree, and the average comprehensive quality score. A standardized multi-center ultrasound evaluation dataset is finally obtained.

[0135] The layered sampling mechanism under the above multi-dimensional quota constraint ensures that the dataset covers the diversity requirements of multi-center, multi-equipment, and multi-disease, strictly controls the number of samples from each source and the overall quality distribution, solves the contradiction between the lack of high-quality samples and the quality requirements through dynamic quota adjustment, and ensures the representativeness and controllability of the evaluation dataset.

[0136] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-center data standardization method for evaluating AI ultrasound model performance, characterized in that, Comprise: S1, collect raw ultrasound image data from different data sources, desensitize and standardize the raw ultrasound image data, and obtain standardized preprocessed images; S2, construct and pretrain a multi-task quality evaluation model, which includes a target region detection module, an integrity scoring module, a clarity classification module, and an artifact detection module; S3, apply the pre-trained multi-task quality evaluation model to quantitatively evaluate all standardized preprocessed images to obtain quality evaluation indicators, and use a batch adaptive threshold strategy and a multi-level screening mechanism based on the quality evaluation indicators to screen the quality, obtaining labeled images with qualified quality; The quality evaluation indicators include integrity scores, clarity classification labels, and spatially weighted artifact occlusion degrees; wherein the spatially weighted artifact occlusion degree is a quantitative indicator obtained by partitioning the target region detection box and assigning different diagnostic importance weights; Partitioning the target region detection box and assigning different diagnostic importance weights include: The target region detection frame is divided into a core region, a transition region and a peripheral region from the center outward in sequence, the core region is defined as a concentric rectangular region with the center as a reference and an area of 50% of the area of the target region detection frame, the transition region is defined as an annular region between the concentric rectangular region with the center as a reference and an area of 75% of the area of the target region detection frame and the core region, and the peripheral region is defined as an annular region between the target region detection frame and the outer boundary frame of the transition region; diagnostic importance weights are respectively given to the core region, the transition region and the peripheral region 、 and , and ; S4, stratified sampling of labeled images with qualified quality to obtain a standardized multi-center ultrasound evaluation dataset.

2. The multi-center data standardization method for evaluating the performance of an AI ultrasound model according to claim 1, characterized in that, Step S1 includes: Collecting raw ultrasound image data from multiple medical centers, different brands and models of ultrasound equipment, and storing the raw ultrasound image data in DICOM format; Extracting image pixel data and imaging parameters from each raw ultrasound image, including device brand, device model, probe type, imaging depth, and gain parameter; Using optical character recognition to identify and remove sensitive information burned on the image, generating a unique anonymized identifier for each image and establishing an anonymized metadata record table that retains device model and imaging parameters; Standardizing the desensitized image, scaling the original image of different sizes to a preset standard size while maintaining the aspect ratio, and filling the blank area generated by the aspect ratio adjustment with the background mean value; Performing grayscale normalization to map the dynamic range of different devices uniformly to the standard grayscale space through linear mapping.

3. The multi-center data standardization method for evaluating the performance of an AI ultrasound model according to claim 1, characterized in that, The multi-task quality evaluation model uses Vision Transformer as the backbone network, divides the input standardized preprocessed image into multiple image block sequences, extracts global semantic features and local texture features through self-attention mechanism; design a target region detection head on the feature map output by the backbone network, and use an anchor-based target detection method to output the target region detection box of the target organ or lesion; the integrity scoring module, the clarity classification module, and the artifact detection module share global semantic features and local texture features for parallel processing.

4. The multi-center data standardization method for evaluating the performance of an AI ultrasound model according to claim 3, characterized in that, The integrity scoring module calculates the intersection over union of the predicted target region detection box and the expert-labeled target region detection box to obtain the integrity score; The sharpness classification module outputs a binary classification label of clear or blurred based on image gradient features and texture features; the artifact detection module outputs a set of artifact region bounding boxes and corresponding artifact type labels, the artifact types including motion artifact, gas artifact, reverberation artifact and acoustic shadow artifact; the multi-task quality assessment model is pre-trained using labeled data from multiple medical centers, the labeled data including expert-labeled target region detection boxes, completeness scores, sharpness labels, and artifact bounding boxes and types.

5. The multi-center data standardization method for evaluating the performance of an AI ultrasound model according to claim 1, characterized in that, The spatially weighted artifact occlusion degree calculation method is: For each detected artifact region bounding box, calculate its intersection area with the target region detection box, count the number of pixels in the intersection area falling in the core region, the transition region and the peripheral region respectively, and calculate the weighted occlusion degree of the single artifact by weighted sum of the pixel number in each region and the corresponding diagnostic importance weight When there are multiple artifacts, merge the intersection areas of all artifact region bounding boxes with the target region detection box to obtain the total occlusion area, count the number of pixels in the total occlusion area falling in the core region, the transition region and the peripheral region respectively to calculate the cumulative weighted occlusion degree .

6. The multi-center data standardization method for evaluating the performance of an AI ultrasound model according to claim 1, characterized in that, The batch-adaptive threshold strategy includes: The standardized preprocessed images are grouped into batches according to data sources, including medical center identifiers and device model identifiers; The distribution characteristics of the quality assessment indicators are calculated for each batch, including the central tendency statistics and dispersion statistics of the completeness score, the proportion of clear samples, the proportion of samples with detected artifacts, and the central tendency statistics and dispersion statistics of the cumulative weighted occlusion degree; Global reference thresholds are pre-set, including a global reference threshold for the completeness score and a global reference threshold for the cumulative weighted occlusion degree; Batch quality deviation is calculated based on the differences between the distribution characteristics of each batch and the global reference thresholds, including completeness deviation, sharpness deviation, and artifact occlusion deviation, which quantify the deviation of the batch relative to the global quality baseline; Based on the differences between the distribution characteristics of each batch and the global reference thresholds, the batches are classified into quality excellent batches, quality medium batches, and quality low batches according to the batch quality deviation; Differentiated judgment thresholds are used for batches of different quality levels: For quality excellent batches, the completeness judgment threshold of the batch is higher than the global reference threshold for the completeness score, and the occlusion degree judgment threshold of the batch is lower than the global reference threshold for the cumulative weighted occlusion degree; For quality low batches, the completeness judgment threshold of the batch is lower than the global reference threshold for the completeness score, and the occlusion degree judgment threshold of the batch is higher than the global reference threshold for the cumulative weighted occlusion degree; For quality medium batches, the global reference threshold is used as the judgment threshold for the batch.

7. The multi-center data standardization method for evaluating the performance of an AI ultrasound model according to claim 6, characterized in that, The multi-level screening mechanism includes: first-level fast pass screening, second-level fine comprehensive evaluation, and third-level manual review and suggestion labeling; In the first-level fast pass screening stage, images that are superior to the batch judgment threshold in the three quality dimensions of completeness score, sharpness classification label, and single artifact maximum weighted occlusion degree are judged as high-quality samples and directly retained; In the second-level fine comprehensive evaluation stage, images that do not pass the first-level screening are comprehensively evaluated. If any of the following conditions is not met: the completeness score is lower than the batch judgment threshold, the sharpness classification label is blurred, or the cumulative weighted occlusion degree exceeds the batch judgment threshold, the image is judged as a unqualified sample and excluded. For images that meet the batch judgment threshold in the three quality dimensions, a comprehensive quality score is calculated. The comprehensive quality score is obtained by weighted fusion of the completeness exceeding degree, the sharpness label, and the occlusion degree redundancy degree. According to the value of the comprehensive quality score, the images are classified into high-quality samples, usable samples, samples to be audited, or unqualified samples. In the third level of artificial review suggestion labeling stage, review suggestion information is generated for images determined to be samples to be audited, including the closeness of each quality dimension to the threshold, the specific location and type of the occlusion, the confidence of the clarity evaluation, and the quality dimensions that the expert should focus on.

8. The multi-center data standardization method for evaluating the performance of an AI ultrasound model according to claim 7, characterized in that, In step S4, the step of stratified sampling of the labeled images of qualified quality includes: A multi-dimensional quota constraint is established, including total sample number constraint, medical center coverage constraint, equipment brand coverage constraint, and quality level distribution constraint; According to the medical center source, equipment brand model, and disease type, the labeled images of qualified quality are stratified, and within each layer, the images are sorted in descending order according to the quality level and the comprehensive quality score, and the images with high quality level and high comprehensive quality score are preferentially selected to fill the quota target number of the layer; For layers with high quality, the target quota number is increased, and for layers with low quality, the target quota number is reduced or the samples to be audited can only be included in the dataset after artificial review; In the process of filling the quota by layer, if the number of high-quality samples in the layer is insufficient, available samples and samples to be audited are supplemented in turn, and the satisfaction of the overall quality level distribution constraint is monitored in real time. If the supplement leads to unsatisfied constraint, the supplement of the layer is stopped and the quota surplus is allocated to other layers; Complete metadata labeling is added to each image finally included in the dataset, and a statistical report of the dataset is established.

Images