AI intelligent medical image labeling system and method based on multi-person cooperation, medium, program product and terminal

Abstract

The invention provides an AI intelligent medical image labeling system and method based on multi-person cooperation, a medium, a program product and a terminal, and the system is characterized in that the system comprises a preprocessing module, a labeling module, a fusion updating module, an expert fine tuning module and a result output module. The system automatically prompts dispute difference labeling results and specifies experts for processing, expert resources are concentrated to solve key problems, and the auditing efficiency and the overall labeling efficiency are improved; according to the method, the credibility model is trained, the labeling credibility of the labeling personnel is accurately identified, the difference labeling result is fused and updated, high-efficiency labeling is achieved, meanwhile, interference of other categories is avoided, the labeling confusion rate is reduced, and the labeling specialty and quality are improved.

Description

Technical Field

[0001] This application relates to the field of medical image annotation technology, and in particular to AI-powered intelligent medical image annotation systems, methods, media, program products, and terminals based on multi-user collaboration. Background Technology

[0002] With the widespread application of artificial intelligence in medical image analysis, the quality of medical image annotation data directly determines the effectiveness of model training and clinical usability. Currently, a large amount of medical image annotation work still relies on manual labor, especially for images with highly complex anatomical structures or lesion types, which often require collaboration among multiple experienced medical experts. Most existing annotation platforms are designed for single-person operation, making it difficult to meet the collaborative annotation needs of high-quality, large-scale, and multimodal medical data.

[0003] Furthermore, in multi-person annotation processes, different annotators often have differing judgments on the same structure or region, lacking effective conflict alerts and coordination mechanisms, leading to chaotic annotation versions and inconsistent quality. This is especially true when multiple structures or categories are involved, where interference between annotators can easily occur, affecting overall annotation efficiency and consistency. Therefore, how to achieve standardization, structuring, and multi-user collaboration in medical image annotation is a pressing issue that needs to be addressed in the construction of medical artificial intelligence training data.

[0004] Traditional medical image annotation methods have the following drawbacks in achieving both high quality and high efficiency in data annotation:

[0005] 1. Low annotation efficiency: Traditional medical image annotation systems are usually operated by a single person, or only allow multiple people to annotate but do not support automatic comparison of differences. Senior experts have limited time and cannot participate in the entire review process, which leads to inconsistent annotations being difficult to detect, ultimately affecting the quality of model learning.

[0006] 2. Difficulty in unifying annotation disputes: In highly specialized medical images, different annotators may have different understandings of boundaries and lesion characteristics, and the existing system lacks a clear decision-making mechanism for the final annotation results. Summary of the Invention

[0007] In view of the shortcomings of the prior art described above, the purpose of this application is to provide an AI-powered intelligent medical image annotation system, method, medium, program product and terminal based on multi-person collaboration, to solve the problems of low annotation efficiency and difficulty in unifying annotation disputes in traditional medical image annotation methods while meeting the requirements of high quality and high efficiency in data annotation.

[0008] To achieve the above and other related objectives, the first aspect of this application provides an AI-powered intelligent medical image annotation system based on multi-person collaboration, comprising: a preprocessing module, an annotation module, a fusion and update module, an expert fine-tuning module, and a result output module; wherein, the preprocessing module is used to acquire medical image data to be annotated, segment the medical image data to be annotated into several regions to be annotated based on a preset annotation task, and assign a corresponding annotation personnel group to each region to be annotated based on a preset annotation personnel database; the annotation analysis module is used to collect the annotation dataset after annotation by the annotation personnel group corresponding to each region to be annotated, and perform difference detection on the annotation dataset corresponding to each region to be annotated, so as to divide the annotation datasets corresponding to several regions to be annotated into a difference annotation dataset and a non-difference annotation dataset; the fusion and update module is used to fuse and update the difference annotation dataset based on a pre-trained credibility model to obtain an updated difference annotation dataset; the expert fine-tuning module is used to perform expert fine-tuning operations on the updated difference annotation dataset to obtain a fine-tuned difference annotation dataset; the result output module is used to integrate the non-difference annotation dataset and the fine-tuned difference annotation dataset to obtain the final annotation dataset of the medical image data to be annotated.

[0009] In some embodiments of the first aspect of this application, the process of performing difference detection on the annotation dataset corresponding to each region to be annotated, so as to divide the annotation datasets corresponding to several regions to be annotated into differential annotation datasets and non-differential annotation datasets, includes: comparing each annotation data in the annotation dataset corresponding to each region to be annotated; if each annotation data in the current region to be annotated is the same, the annotation dataset corresponding to the current region to be annotated is a non-differential annotation dataset; if each annotation data in the current region to be annotated is different, the annotation dataset corresponding to the current region to be annotated is a differential annotation dataset.

[0010] In some embodiments of the first aspect of this application, the training process of the pre-trained credibility model includes: initializing the credibility parameters of the annotators in each annotator group in the preset annotator database to obtain the initial credibility of each annotator and constructing an initial credibility model; acquiring a medical image training dataset and the real-time annotation results of each annotator in the preset annotator database on the medical image training dataset; performing a weighted operation on the initial credibility of each annotator based on the real-time annotation results of the medical image training dataset to obtain a weighted initial credibility; comparing the real-time annotation results of the medical image training dataset with the standard annotation results of the medical image training dataset, updating and weighting the weighted initial credibility in the initial credibility model until the initial credibility model converges to obtain the pre-trained credibility model.

[0011] In some embodiments of the first aspect of this application, the process of performing a weighted operation on the initial confidence of each annotator based on the real-time annotation results of the medical image training dataset to obtain the weighted initial confidence includes: obtaining a weighted fusion annotation result based on the real-time annotation results corresponding to each annotator in each annotator group and the initial confidence using a weighted average function; calculating the first annotation error of each annotator based on the weighted fusion annotation result of each annotator group and the real-time annotation results corresponding to each annotator in each annotator group using a loss function; and normalizing the initial confidence based on the first annotation error of each annotator using a normalization function to obtain the weighted initial confidence corresponding to each annotator.

[0012] In some embodiments of the first aspect of this application, the process of updating the weighted initial credibility in the initial credibility model by comparing the real-time annotation results of the medical image training dataset with the standard annotation results of the medical image training dataset includes: calculating the second annotation error corresponding to each annotator using a loss function based on the real-time annotation results and the standard annotation results of the medical image training dataset; calculating the average value after the cumulative number of calculations of the second annotation error of each annotator reaches a preset number of records to obtain the target annotation error corresponding to each annotator; calculating the credibility update function based on the target annotation error corresponding to each annotator, and updating the weighted initial credibility according to the calculation results.

[0013] In some embodiments of the first aspect of this application, the process of fusing and updating the differential annotation dataset based on a pre-trained credibility model includes: comparing each annotation result in the differential annotation dataset within a group to obtain the intra-group difference value corresponding to each annotator; filtering the intra-group difference values ​​corresponding to each annotator to obtain annotators whose intra-group difference values ​​are less than or equal to a preset threshold; and calculating the updated differential annotation dataset based on the credibility of the selected annotators and the annotation results using a weighted average function.

[0014] To achieve the above and other related objectives, a second aspect of this application provides an AI-powered intelligent medical image annotation method based on multi-person collaboration. Applied to the aforementioned AI-powered intelligent medical image annotation system based on multi-person collaboration, the method includes: acquiring medical image data to be annotated; dividing the medical image data into several regions to be annotated based on a preset annotation task; assigning an annotation team to each region to be annotated based on the professional qualifications of the annotation team; collecting annotation data generated by the annotation team after annotating the assigned regions to be annotated, to obtain annotation data groups with identical annotation data and differential annotation data groups with slightly different annotation data within the regions to be annotated; calculating the differential annotation data groups based on a credibility model to obtain an updated differential annotation data group; analyzing the updated differential annotation data group based on the differential annotation data group to obtain an unchanged updated differential annotation data group or a finely adjusted differential annotation data group; and confirming the annotation data group, the updated differential annotation data group, and the finely adjusted differential annotation data group to obtain the final annotation data for the regions to be annotated.

[0015] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the AI-based intelligent medical image annotation method based on multi-person collaboration.

[0016] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code, which, when executed on a computer, enables the computer to implement the AI-based intelligent medical image annotation method based on multi-person collaboration.

[0017] To achieve the above and other related objectives, a fifth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the AI-based intelligent medical image annotation method based on multi-person collaboration.

[0018] As described above, the AI-powered intelligent medical image annotation system, method, medium, program product, and terminal based on multi-person collaboration of this application have the following beneficial effects:

[0019] (1) This application constructs a multi-person collaborative image annotation system, compares the results of multiple annotators, the system automatically prompts the disputed difference annotation results and assigns experts to handle them, concentrates expert resources to solve key problems, and improves the efficiency of review and overall annotation.

[0020] (2) This application trains a credibility model to accurately identify the credibility of the annotation by the annotator, merges and updates the different annotation results, and avoids interference from other categories while efficiently annotating, reducing the annotation confusion rate and improving the professionalism and quality of annotation. Attached Figure Description

[0021] Figure 1 The diagram shown is a structural schematic of an AI-powered intelligent medical image annotation system based on multi-person collaboration, as described in one embodiment of this application.

[0022] Figure 2 The diagram shown is an example of an annotation task for an AI-powered intelligent medical image annotation system based on multi-person collaboration, as described in one embodiment of this application.

[0023] Figure 3 The diagram shows the workflow of an AI-powered intelligent medical image annotation system based on multi-person collaboration, as described in one embodiment of this application.

[0024] Figure 4 The diagram shown is a flowchart of an AI-powered intelligent medical image annotation method based on multi-person collaboration in one embodiment of this application.

[0025] Figure 5 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0026] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0027] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," and "holding" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0028] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0029] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, "first annotation error" and "second annotation error" are only used to distinguish different annotation error data and do not limit their order. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that terms such as "first" and "second" do not necessarily imply that they are different.

[0030] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the invention.

[0031] like Figure 1 As shown, this embodiment of the invention provides an AI-powered intelligent medical image annotation system based on multi-person collaboration, used to annotate medical images. The system 100 includes: a preprocessing module 101, an annotation module 102, a fusion and update module 103, an expert fine-tuning module 104, and a result determination module 105.

[0032] The preprocessing module 101 is used to acquire medical image data to be labeled, divide the medical image data to be labeled into several regions to be labeled based on a preset labeling task, and assign a corresponding labeling personnel group to each region to be labeled based on a preset labeling personnel database.

[0033] The medical image data to be labeled refers to raw medical image data that has not yet undergone labeling processing. This medical image data may include computed tomography (CT) images, magnetic resonance imaging (MRI) images, ultrasound images, or X-ray images. This type of data is typically acquired by medical imaging equipment and contains anatomical structural information and potential lesion information related to the examined subject, but does not include labeled results for training or validating medical image analysis models. Preferably, during implementation, the medical image data to be labeled can undergo preprocessing operations such as format standardization, noise suppression, and grayscale normalization to lay the foundation for subsequent labeling work.

[0034] It should be noted that the preset annotation task includes anatomical structure, image region, and annotation type. The annotation type includes, but is not limited to, image segmentation, key point recognition, lesion recognition, etc. The annotation type is selected according to the actual annotation needs and is not specifically limited here.

[0035] Furthermore, the preset annotation personnel database contains multiple annotation personnel groups. Each annotation personnel group stores the personal information, professional annotation areas, and professional annotation types of each annotation personnel. The professional annotation areas and professional annotation types represent the annotation areas and types that the corresponding annotation personnel are most proficient in.

[0036] It should be noted that the number of annotators within each annotator group is not fixed, and the number of annotators varies between different annotator groups. For example, annotator group 1 may contain three annotators, annotator group 2 may contain three annotators, and annotator group 3 may contain five annotators. The number of annotators should be set according to the capabilities of each annotator and actual needs.

[0037] Furthermore, by matching the capabilities of the annotation personnel groups in the annotation personnel database with the location and annotation type of the area to be annotated, a corresponding annotation personnel group can be assigned to each area to be annotated.

[0038] Furthermore, for example, such as Figure 2As shown, for example, if the medical image data to be labeled is a CT scan image of a human skeleton, the complete human skeleton CT scan image is first segmented into different sub-regions based on anatomical structure, such as the head region, trunk region, and limb regions. Then, target regions that meet the image region requirements are selected from the head region, trunk region, and limb regions, such as the pelvic region, femur region, and leg bone region. Finally, the pelvic region, femur region, and leg bone region, combined with the annotation type, will be defined as the region to be labeled. The same target region can be divided into different regions to be labeled by different annotation types. For example, if the annotation type includes image segmentation and key point recognition, then based on the two annotation types and the three target regions of pelvic region, femur region, and leg bone region, six regions to be labeled will be obtained: image segmentation pelvic region, image segmentation femur region, image segmentation leg bone region, key point recognition pelvic region, key point recognition femur region, and key point recognition leg bone region. Figure 2 Taking key point recognition of the pelvic region, key point recognition of the femur region, and key point recognition of the leg bone region as examples, corresponding to three sub-tasks: sub-task 1, sub-task 2, and sub-task 3, the most suitable annotation personnel group is matched for the three sub-tasks based on the personal information, professional annotation area, and professional annotation type of each annotator in each annotation personnel group in the preset annotation personnel database, so as to carry out the subsequent annotation tasks.

[0039] Collaborative annotation of medical images, leveraging the capabilities of multiple annotators, significantly improves data quality while maintaining efficiency. The parallel work and cross-checking of multiple annotators not only increases annotation speed but also effectively reduces subjective bias caused by individual annotation. The complementary professional backgrounds and knowledge structures of different annotators ensure more comprehensive, accurate, and consistent annotation results. Furthermore, the collaborative annotation process establishes unified annotation standards and quality control mechanisms, ensuring higher reliability and reusability of the dataset, thus providing a high-quality data foundation for subsequent model training or technical analysis.

[0040] The annotation analysis module 102 is used to collect the annotation dataset after annotation by the annotation personnel group for each region to be annotated, and to perform difference detection on the annotation dataset corresponding to each region to be annotated, so as to divide the annotation datasets corresponding to several regions to be annotated into differential annotation datasets and non-differential annotation datasets.

[0041] Preferably, the human-computer interface (HCI) of the AI-powered intelligent medical image annotation system is used by annotators. This HCI includes a display screen and input devices to annotate each assigned region. The HCI supports multi-user collaborative annotation of medical image data. Specifically, the HCI supports parallel assignment and collaborative work among multiple annotators to improve the annotation efficiency of large-scale medical image data. The HCI is equipped with structured annotation tools to provide diverse interaction methods for different types of annotation needs, including but not limited to segmentation brushes, keypoint frames, and 3D view displays, to meet different image structures and annotation accuracy requirements.

[0042] Furthermore, during the annotation process, the human-computer interface can record the operation trajectory, annotation time, and editing version information of each annotator in real time, facilitating subsequent version tracking, annotation quality evaluation, and discrepancy detection. In addition, the human-computer interface also features a region locking mechanism to prevent multiple annotators from simultaneously annotating the same area, thereby avoiding data conflicts and annotation overwrite issues. Through these designs, the system can achieve efficient allocation, precise management, and process traceability of annotation tasks, ensuring the integrity and consistency of annotation data.

[0043] In one embodiment, the process of performing difference detection on the annotation dataset corresponding to each region to be annotated to divide the annotation datasets corresponding to several regions to be annotated into differential annotation datasets and non-differential annotation datasets includes: comparing each annotation data in the annotation dataset corresponding to each region to be annotated; if each annotation data in the current region to be annotated is the same, the annotation dataset corresponding to the current region to be annotated is a non-differential annotation dataset; if each annotation data in the current region to be annotated is different, the annotation dataset corresponding to the current region to be annotated is a differential annotation dataset.

[0044] For example, if a group of three annotators annotates a region to be annotated, the resulting annotation dataset contains three annotation results from each of the three annotators. If the three annotation results are completely identical to each other, the annotation dataset corresponding to the current annotator group is determined to be a non-discrepant annotation dataset. If at least one of the three annotation results is different, the annotation dataset corresponding to the current annotator group is determined to be a discrepant annotation dataset.

[0045] It is important to emphasize that the annotation results must be completely identical pairwise. To improve annotation redundancy, an error range is set based on actual annotation conditions. Specifically, two annotation results are considered identical if the difference is within a preset difference value. The preset difference value is set to a low value based on factors such as the image quality of the actual region to be annotated. For example, in image segmentation, the difference in the area of ​​the segmented region may be within 0.5%, or in keypoint recognition, the distance between keypoint coordinates may be within 0.1% of the width of the region to be annotated. Specific values ​​can be set as needed; no specific limitations are imposed here.

[0046] The fusion update module 103 is used to fuse and update the differential labeling dataset based on the pre-trained credibility model to obtain the updated differential labeling dataset.

[0047] In one embodiment, the training process of the pre-trained credibility model includes: initializing the credibility parameters of the annotators in each annotator group in the preset annotator database to obtain the initial credibility of each annotator and constructing an initial credibility model; acquiring a medical image training dataset and the real-time annotation results of each annotator in the preset annotator database on the medical image training dataset; performing a weighted operation on the initial credibility of each annotator based on the real-time annotation results of the medical image training dataset to obtain a weighted initial credibility; comparing the real-time annotation results of the medical image training dataset with the standard annotation results of the medical image training dataset, updating and weighting the weighted initial credibility in the initial credibility model until the initial credibility model converges to obtain the pre-trained credibility model.

[0048] It should be noted that the credibility parameter of each annotator in the preset annotator database is initialized by calculating the historical consistency score. The historical consistency score is obtained by comparing the annotation results of each annotator with those of other annotators in the same annotator group in historical annotation tasks. For example, if an annotator group contains annotator 1, annotator 2, and annotator 3, and in one image annotation, annotator 1's annotation content is identical to that of annotators 2 and 3, then this image annotation is considered consistent image annotation. If annotator 1's annotation content is not entirely identical to that of annotators 2 and 3, then this image annotation is considered inconsistent image annotation. The proportion of consistent image annotations by annotator 1 in several historical image annotations to the total number of image annotations is the historical consistency score of annotator 1. The total number of image annotations is not limited here as needed. The initial credibility is used to initially measure the current annotator's annotation ability. The historical consistency score can also reflect the annotator's recent annotation ability, and the historical consistency score ranges from 0 to 1. Therefore, the historical consistency score of annotator 1 is set as the initial credibility of annotator 1, and the initial credibility range is also 0 to 1.

[0049] Furthermore, once the credibility parameters of the annotators in each annotator group in the preset annotator database are initialized, a credibility model containing the personal information and initial credibility of the annotators in each annotator group is constructed.

[0050] The medical image training dataset is used to train the credibility model of the annotators. This dataset comes from historical medical image data that has been collaboratively annotated by multiple annotators. The medical image training dataset includes standard annotation results and original medical images, with each original medical image corresponding to a standard annotation result.

[0051] Furthermore, each annotator in the pre-defined annotator group annotates the original medical images in the medical image training dataset. The system will collect the real-time annotation results of the annotator group on the original medical images, where each real-time annotation result includes the annotation results of all annotators in an annotator group on an original medical image.

[0052] In one embodiment, the process of weighting the initial confidence of each annotator based on the real-time annotation results of the medical image training dataset to obtain the weighted initial confidence includes: obtaining a weighted fused annotation result based on the real-time annotation results corresponding to each annotator in each annotator group and the initial confidence using a weighted average function; calculating the first annotation error of each annotator based on the weighted fused annotation result of each annotator group and the real-time annotation results corresponding to each annotator in each annotator group using a loss function, wherein the first annotation error represents the annotation error of each annotator in an annotator group compared with other annotators in the same annotated region; and normalizing the initial confidence based on the first annotation error of each annotator using a normalization function to obtain the weighted initial confidence corresponding to each annotator.

[0053] To maintain the relative proportion of credibility and ensure the comparability of the total credibility within a group, a weighted normalization operation is performed on the initial credibility of each annotator based on the real-time annotation results of the original image data from each annotator within the annotator group. This normalization operation ensures that the overall numerical range of credibility among members remains within a preset range (e.g., [0,1]), thus avoiding numerical divergence or excessive amplification. Furthermore, the credibility distribution of each group after normalization is under a unified standard, facilitating subsequent result fusion, weight calculation, and model parameter sharing between different annotator groups. Therefore, by performing a weighted operation after updating within the group, a stable constraint on the credibility distribution can be achieved while maintaining individual differences, thereby ensuring the training consistency and fusion effect of the annotator credibility model in multi-group, multi-annotation scenarios.

[0054] First, a weighted fusion annotation result is obtained by using a weighted average function based on the real-time annotation results of each annotator in the annotator group and their initial confidence level. , The calculation formula is shown in Formula 1:

[0055] ;(Formula 1)

[0056] in This indicates the initial credibility of each annotator within the annotator group. This indicates the relationship between each annotator in the annotator group and... The corresponding real-time annotation results.

[0057] Secondly, based on the weighted fusion annotation results of each annotator group Real-time annotation results for each annotator within each annotator group. The first annotation error of each annotator is calculated using a loss function. .

[0058] Preferably, the DiceLoss function is used as the preferred loss function. The calculation formula is shown in Formula 2:

[0059] ;(Formula 2)

[0060] in, This indicates the real-time annotation results for each annotator within the annotation team. express The weighted fusion annotation results do not include the portion labeled by annotator i. The calculation formula is shown in Formula 3:

[0061] ;(Formula 3)

[0062] Finally, the initial confidence level is normalized using a normalization function to obtain the weighted initial confidence level for each annotator. initial credibility after weighting The calculation formula is shown in Formula 4:

[0063] ;(Formula 4)

[0064] in This represents the initial confidence level of annotator i after the nth normalization. This indicates the initial sensitivity of each annotator within the annotator group. For sensitivity parameters, You can set it as needed; there are no specific restrictions here.

[0065] In one embodiment, the process of updating the weighted initial credibility in the initial credibility model by comparing the real-time annotation results of the medical image training dataset with the standard annotation results of the medical image training dataset includes: calculating the second annotation error corresponding to each annotator using a loss function based on the real-time annotation results and the standard annotation results of the medical image training dataset, where the second annotation error represents the error between the annotator's annotation result and the standard annotation result; calculating the average value after the cumulative number of calculations of the second annotation error of each annotator reaches a preset number of records to obtain the target annotation error corresponding to each annotator; and updating the weighted initial credibility based on the target annotation error corresponding to each annotator using a credibility update function.

[0066] Specifically, the second annotation error is calculated using a loss function between the real-time annotation results of the annotators on the medical image training dataset and the standard annotation results on the same dataset. This second annotation error represents the error between the annotators' current annotation results and the actual results. The DiceLoss function is preferred among the loss functions used. The calculation formula is shown in Formula 5:

[0067] ;(Formula 5)

[0068] in This represents the real-time annotation results of each annotator within the annotation team; GT indicates the sum of all annotations in the data. The corresponding standard annotation results.

[0069] Furthermore, after calculating the second annotation error for the current time, the second annotation error is saved to the error record of the corresponding annotator. Then, a cumulative judgment is performed. When the number of second annotation errors saved in the error record of the corresponding annotator exceeds a preset number W, the average value of the second annotation errors saved in the error record of the annotator is calculated to obtain the target annotation error of the corresponding annotator. And clear the error records of the corresponding labelers. The calculation is shown in Formula 6:

[0070] ;(Formula 6)

[0071] Where W represents the preset number of records. The setting of W can be adjusted based on the total number of annotations as needed, and no specific limitation is made here. Indicates the target annotation error. The second annotation error is the i-th annotation.

[0072] Finally, based on the target annotation error, the weighted confidence is updated using the confidence update function to obtain the updated confidence function. , The calculation formula is shown in Formula 7:

[0073] ;(Formula 7)

[0074] in Let be the second annotation error of the i-th annotation, where j represents the j-th confidence update, and k represents the number of annotators in the annotation group to which the current annotator i belongs. This represents the second annotation error of the k-th annotator. This indicates that the parameters are being updated. The value is set as needed and >1.

[0075] It should be emphasized that each training process is conducted in groups of annotators. Therefore, all annotators in the annotator group in Formula 7 will simultaneously meet the preset number of records W.

[0076] Specifically, by calculating the error between the real-time annotation results of the annotators on the original medical images in the medical image training dataset and the standard annotation results, the accuracy of the annotators' annotations is determined. This dynamically updates the initial weighted credibility of the annotators in the credibility model. The initial weighted credibility of all annotators in the credibility model is continuously and dynamically updated until the initial weighted credibility of all annotators corresponds to the annotation ability of the annotators themselves, and the training of the credibility model is completed.

[0077] Furthermore, when the credibility updates of all annotators within all annotator groups remain unchanged within a preset stable number of times, it can be determined that the weighted initial credibility of all annotators corresponds to the annotator's own annotation ability, that is, the training of the credibility model is completed. The preset stable number of times can be set according to training needs, and no specific limitation is made here.

[0078] By training a credibility model, the credibility of different annotators in multi-user annotated data can be quantitatively evaluated, and their credibility can be dynamically updated during training based on the consistency between their annotation results and the standard annotation results. In subsequent annotation tasks, this credibility model can weight the annotation results according to the credibility of the annotators, thereby effectively reducing the impact of low-quality annotations on the overall dataset quality. Compared with traditional averaging or single-person judgment methods, the credibility model-based annotation mechanism can significantly improve the accuracy and consistency of data annotation, enhance the objectivity and interpretability of annotation results, and ultimately obtain a higher quality and more stable dataset, providing more reliable data support for subsequent model training and intelligent analysis.

[0079] In one embodiment, the process of fusing and updating the differential annotation dataset based on a pre-trained credibility model includes: comparing each annotation result in the differential annotation dataset within a group to obtain the intra-group difference value corresponding to each annotator; filtering the intra-group difference values ​​corresponding to each annotator to obtain annotators whose intra-group difference values ​​are less than or equal to a preset threshold; and calculating the updated differential annotation dataset based on the credibility of the selected annotators and the annotation results using a weighted average function.

[0080] For example, suppose a group of annotators has three annotators. After annotating a region to be annotated, difference detection is performed to obtain the difference annotation data for that region. First, the intra-group difference value is calculated. Depending on the annotation type in the preset annotation task, the calculation method for the intra-group difference value varies. For example, keypoint detection can use distance-based methods such as Euclidean distance and Manhattan distance; image segmentation uses methods based on geometric overlap, such as the IoU difference method. The specific method used is selected as needed and is not specifically limited here. Then, after obtaining the intra-group difference values ​​for each of the three annotators, if the intra-group difference value is greater than a preset threshold, the annotation results of the annotator corresponding to that intra-group difference value will be removed from the difference annotation data. The preset threshold is set as needed based on the annotation type and the corresponding intra-group difference value calculation method and is not specifically limited here. Finally, a weighted average function is used to calculate the updated difference annotation data based on the confidence level and the annotation results within the removed difference annotation data. The calculation of the updated difference annotation data is shown in Formula 8.

[0081] ;(Formula 8)

[0082] Where Q represents the number of annotators in the annotator group.

[0083] The expert fine-tuning module 104 is used to perform expert fine-tuning operations on the updated differential annotation dataset to obtain the fine-tuned differential annotation dataset.

[0084] like Figure 3 As shown, in order to ensure that the final annotation results meet medical standards and quality requirements, the updated differential annotation dataset in the system will be fine-tuned by experts as a final manual check and correction operation, thereby generating more accurate and consistent annotation results. This not only improves data quality, but also provides accurate supervision signals for subsequent model training.

[0085] The result output module 105 is used to integrate the non-differential labeled dataset and the fine-tuned differential labeled dataset to obtain the final labeled dataset of the medical image data to be labeled.

[0086] like Figure 4 The diagram illustrates an AI-powered intelligent medical image annotation method based on multi-user collaboration, as described in one embodiment of the present invention. This method is applied to the AI-powered intelligent medical image annotation system based on multi-user collaboration as described above. The specific steps are as follows:

[0087] Step S41: Obtain the medical image data to be labeled, divide the medical image data to be labeled into several regions to be labeled based on the preset labeling task, and assign a labeling personnel group to each region to be labeled based on the professional qualifications of the labeling personnel group.

[0088] Step S42: Collect the annotation data generated by the annotation personnel group after annotating the assigned areas to be annotated, so as to obtain the annotation data groups with the same annotation data and the differential annotation data groups with different annotation data within the areas to be annotated;

[0089] Step S43: Calculate the difference-annotated data set based on the credibility model to obtain an updated difference-annotated data set;

[0090] Step S44: Analyze the updated difference labeling data group based on the difference labeling data group to obtain an unchanged updated difference labeling data group or a finely adjusted difference labeling data group.

[0091] Step S45: Confirm the labeled data group, the updated difference labeled data group, and the fine-tuned difference labeled data group to obtain the final labeled data for the region to be labeled.

[0092] Figure 5 This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 5 As shown, the computer device includes at least one processor 501, a memory 502, at least one network interface 503, and a user interface 505. The various components in the device are coupled together via a bus system 504. It is understood that the bus system 504 is used to implement communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 5 The general will label all buses as bus systems.

[0093] The user interface 505 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0094] It is understood that memory 502 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0095] In this embodiment of the invention, the memory 502 is used to store various types of data to support the operation of the electronic terminal 500. Examples of this data include: any executable program for operation on the electronic terminal 500, such as the operating system 5021 and application program 5022; the operating system 5021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 5022 may contain various applications, such as media players, browsers, etc., for implementing various application services. The implementation of the AI-based intelligent medical image annotation method based on multi-person collaboration provided in this embodiment of the invention can be included in the application program 5022.

[0096] The methods disclosed in the above embodiments of the present invention can be applied to processor 501, or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 501 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 501 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0097] In an exemplary embodiment, the electronic terminal 500 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0098] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute the AI-based intelligent medical image annotation method based on multi-person collaboration in any of the embodiments shown.

[0099] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to execute the AI-based intelligent medical image annotation method based on multi-person collaboration in any of the embodiments shown.

[0100] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0101] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0102] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0104] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0106] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0107] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0109] In summary, this application provides an AI-powered intelligent medical image annotation system, method, medium, program product, and terminal based on multi-person collaboration. Its features include: a preprocessing module, an annotation module, a fusion and update module, an expert fine-tuning module, and a result output module. This application constructs a multi-person collaborative image annotation system that compares the results of multiple annotators. The system automatically identifies "disagreement areas" and assigns priority processing to experts, concentrating expert resources to solve key problems and improving review and overall annotation efficiency. By training a credibility model, the system accurately identifies the credibility of annotators' annotations, merges and updates differing annotation results, efficiently annotating while avoiding interference from other categories, reducing annotation confusion rates, and improving annotation professionalism and quality. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0110] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A multi-user collaborative AI-powered intelligent medical image annotation system, characterized in that: include: It includes a preprocessing module, a labeling module, a fusion and update module, an expert fine-tuning module, and a results output module; The preprocessing module is used to acquire medical image data to be labeled, divide the medical image data to be labeled into several regions to be labeled based on a preset labeling task, and assign a corresponding labeling personnel group to each region to be labeled based on a preset labeling personnel database. The annotation analysis module is used to collect the annotation dataset after the annotation is performed by the annotation personnel group for each region to be annotated, and to perform difference detection on the annotation dataset corresponding to each region to be annotated, so as to divide the annotation datasets corresponding to several regions to be annotated into differential annotation datasets and non-differential annotation datasets. The fusion update module is used to fuse and update the differentially labeled dataset based on the pre-trained credibility model to obtain the updated differentially labeled dataset. The expert fine-tuning module is used to perform expert fine-tuning operations on the updated differential annotation dataset to obtain the fine-tuned differential annotation dataset. The result output module is used to integrate the non-differential labeled dataset and the fine-tuned differential labeled dataset to obtain the final labeled dataset of the medical image data to be labeled.

2. The AI-powered intelligent medical image annotation system based on multi-person collaboration according to claim 1, characterized in that, The process of performing difference detection on the labeled dataset corresponding to each region to be labeled, so as to divide the labeled datasets corresponding to several regions to be labeled into differentially labeled datasets and non-differentially labeled datasets, includes: Compare each labeled data in the labeled dataset corresponding to each region to be labeled; If all the labeled data in the current region to be labeled are the same, the labeled dataset corresponding to the current region to be labeled is a non-differential labeled dataset; If the annotation data in the current region to be annotated are different, the annotation dataset corresponding to the current region to be annotated is a differential annotation dataset.

3. The AI-powered intelligent medical image annotation system based on multi-person collaboration according to claim 1, characterized in that, The training process of the pre-trained credibility model includes: Initialize the credibility parameters of the annotators in each annotator group in the preset annotator database to obtain the initial credibility of each annotator and build the initial credibility model; Obtain the medical image training dataset, as well as the real-time annotation results of each annotator in the preset annotator database on the medical image training dataset; Based on the real-time annotation results of the medical image training dataset, a weighted operation is performed on the initial confidence of each annotator to obtain the weighted initial confidence. Based on the comparison between the real-time annotation results of the medical image training dataset and the standard annotation results of the medical image training dataset, the weighted initial confidence in the initial confidence model is updated and weighted until the initial confidence model converges, thus obtaining the pre-trained confidence model.

4. The AI-powered intelligent medical image annotation system based on multi-person collaboration according to claim 3, characterized in that, The process of weighting the initial confidence of each annotator based on the real-time annotation results of the medical image training dataset to obtain the weighted initial confidence includes: The weighted fusion annotation result is obtained by using a weighted average function based on the real-time annotation results of each annotator in each annotator group and the initial confidence level. Based on the weighted fusion annotation results of each annotator group and the real-time annotation results of each annotator within each annotator group, the first annotation error of each annotator is calculated using a loss function. The initial confidence level is normalized using a normalization function based on the first annotation error of each annotator, resulting in a weighted initial confidence level for each annotator.

5. The AI-powered intelligent medical image annotation system based on multi-person collaboration according to claim 3, characterized in that, The process of updating the weighted initial confidence level in the initial confidence level model by comparing the real-time annotation results of the medical image training dataset with the standard annotation results of the medical image training dataset includes: The second annotation error for each annotator is calculated using a loss function based on the real-time annotation results of the medical image training dataset and the standard annotation results of the medical image training dataset. Once the cumulative number of times the second annotation error of each annotator is calculated reaches the preset number of records, the average value is calculated to obtain the target annotation error corresponding to each annotator. The confidence level is calculated using a confidence update function based on the target annotation error of each annotator, and the weighted initial confidence level is updated based on the calculation results.

6. The AI-powered intelligent medical image annotation system based on multi-person collaboration according to claim 1, characterized in that, The process of fusing and updating the differentially labeled dataset based on a pre-trained credibility model includes: The annotation results in the differential annotation dataset are compared within groups to obtain the intra-group difference value for each annotator. The intra-group variance values ​​for each annotator are filtered to obtain annotators whose intra-group variance values ​​are less than or equal to a preset threshold. A weighted average function is then used to calculate the updated variance annotation dataset based on the confidence level of the selected annotators and the annotation results.

7. A multi-person collaborative AI-powered intelligent medical image annotation method, characterized in that, The method, applied to the AI-based multi-person collaborative medical image annotation and training system as described in any one of claims 1 to 6, comprises: Acquire medical image data to be labeled, divide the medical image data to be labeled into several regions to be labeled based on a preset labeling task, and assign a labeling personnel group to each region to be labeled based on the professional qualifications of the labeling personnel group; The annotation data generated by the annotation personnel group after annotating the assigned areas to be annotated is collected, so as to obtain the annotation data groups with the same annotation data and the differential annotation data groups with different annotation data within the areas to be annotated; The updated differentially labeled data set is obtained by calculating based on the credibility model. The updated difference labeling data set is analyzed based on the difference labeling data set to obtain an unchanged updated difference labeling data set or a finely adjusted difference labeling data set. The labeled data group, the updated difference labeled data group, and the fine-tuned difference labeled data group are confirmed to obtain the final labeled data for the region to be labeled.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the AI-based intelligent medical image annotation method based on multi-person collaboration as described in claim 7.

9. A computer program product, characterized in that, The computer program product includes computer program code, which, when run on a computer, enables the computer to implement the AI-based intelligent medical image annotation method based on multi-person collaboration as described in claim 7.

10. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the AI-based intelligent medical image annotation method based on multi-person collaboration as described in claim 7.

Images