Observer sequential bias mitigation in medical imaging
By employing a deep learning model to estimate and mitigate pairwise image similarity, the method reorganizes medical image sequences to minimize observer bias, enhancing diagnostic accuracy in medical imaging systems, addressing the issue of serial dependence and improving sensitivity and specificity in telemedicine-based diagnoses.
Patent Information
- Application Number
- PCT/US2024/029313
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-15
- Filing Date
- 2024-05-14
- Publication Date
- 2024-11-21
AI Technical Summary
Medical imaging systems, particularly in telemedicine, suffer from observer bias due to serial dependence, leading to systematic errors in diagnostic judgments, such as reduced sensitivity and specificity, and increased error rates in tasks like skin cancer diagnosis.
A method using a deep learning model to estimate pairwise image similarity and reorganize image sequences to minimize sequential bias, maximizing differences between images to reduce observer-specific idiosyncratic errors and improve diagnostic accuracy.
The solution significantly improves the accuracy of medical image judgments by reducing sequential biases, enhancing sensitivity, specificity, and reducing error rates in medical imaging tasks, including telemedicine-based diagnoses.
Smart Images

Figure US2024029313_21112024_PF_FP_ABST
Abstract
Description
OBSERVER SEQUENTIAL BIAS MITIGATION IN MEDICAL IMAGINGRELATED APPLICATIONS
[0001] This disclosure is a non-provisional of and claims benefit from US ProvisionalApplication No. 63 / 502,348, titled “OBSERVER BIAS MITIGATION IN MEDICALIMAGING,” filed on May 15, 2023, the disclosure of which is incorporated herein by reference in its entirety.STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT
[0002] This invention was made with Government support under award numberR01CA236793 awarded by the National Institutes of Health. The Government has certain rights in this invention.TECHNICAL FIELD
[0003] This disclosure relates to batch processing of images in a medical environment, more particularly to removal of observer bias.BACKGROUND
[0004] Research has identified sequential effects, such as serial dependence, as a cause of systematic judgment errors in medical imaging systems as disclosed in Manassi, Mauro, et al."Serial dependence in the perceptual judgments of radiologists." Cognitive research: principles and implications 6 (2021): Manassi, Mauro, Arni Kristjansson, and DavidWhitney. "Serial dependence in a simulated clinical visual search task." Scientific reports 9.1(2019): 19937; and Fischer, Jason, and David Whitney. "Serial dependence in visual perception." Nature neuroscience 17.5 (2014): 738-743. Mannassi, Mauro, et al. discussessequential bias in mammography not teledermatology, indicating that the bias happens in many domains.
[0005] This may include remote store-and-forward telemedicine systems. These systems generally gather information including images from a patient, store them, and then forward them to a medical provider for analysis across one or more networks.
[0006] Other systems include any system that performs batch processing of images. These may include hospitals or other medical institutions where the user receives imaging services, and the system stores the images for later processing or analysis by a provider.
[0007] The natural visual world is autocorrelated: objects do not spontaneously pop in to or out of existence in the typical visual experience, and what was present a moment ago tends to still be present at this moment. The human visual system has developed adaptive mechanisms that take advantage of these natural autocorrelations by introducing serial dependence in perceptual interpretations. Due to this mechanism, objects recognized at one moment appear more like similar objects seen in the last several seconds. The result of this serial dependence is that perceptual experience seems smoother and more stable than it should be. This is beneficial because without it, the visual world would look jittery and unstable; object identities would appear to fluctuate due to changes in lighting, viewpoint, blinks, and myriad sources of internal and external noise.
[0008] It is intuitive that human vision benefits from recycling visual history, smoothing and stabilizing perceptual experience in the natural world. However, the benefit of serial dependence has limits because the visual world is not always natural. In certain artificial, human-designed visual tasks, such as medical image perception or randomized laboratory experiments, visual stimuli are no longer naturally autocorrelated. Visual images in these situations can vary randomly from one moment to the next. If the visual system imposesserial dependence, smoothing or reusing previous visual history, this could introduce systematic errors by attracting current perception towards previous visual history.
[0009] Studies have shown that this is exactly what happens. Serial dependence systematically biases current perception toward visual history in many tasks, such as perception of orientation, attractiveness, and emotional expression. Serial dependence also introduces systematic perceptual errors in medical image perception tasks. However, these studies were conducted under lab conditions with highly artificial stimuli and experimental designs that are not typical in clinical practice. More recently, progress in GenerativeAdversarial Networks (GANs) affords the opportunity to generate more realistic simulated medical images as stimuli. However, even in these studies, the relatively complex psychophysical tasks were not comparable to realistic, clinically relevant scenarios.BRIEF DESCRIPTION OF THE DRAWINGS
[0010] FIG. 1 shows an embodiment of a medical imaging system having a bias mitigation system in a remote environment.
[0011] FIG. 2 shows an embodiment of a medical imaging system having a bias mitigation system in a local environment.
[0012] FIG. 3 shows an embodiment of a computing device containing a bias mitigation system.
[0013] FIG. 4 shows a flowchart of an embodiment of a method of reorganizing a batch of medical images to mitigate serial dependence.DETAILED DESCRIPTION OF THE EMBODIMENTS
[0014] The embodiments address the effects of serial dependence in image analysis in the medical environment. One example involves remote store-and-forward medical imaging, which involves sequential judgments of static images, is an especially fast-growing area of telemedicine. It requires the involvement of clinicians because automated systems are notsufficient to make accurate diagnostic classifications. The images may comprise radiological images, X-rays, mammography, CT scans, MRIs, etc.
[0015] A study was performed that analyzed 758,139 skin cancer diagnostic judgments from1137 participants collected from an app developed by Centaur Labs, a US medical ArtificialIntelligence (Al) company based in Boston. The task was a straightforward 2AFC (two- alternative forced choice) (yes / no) discrimination, with a goal of diagnosing whether an actual skin cancer image was nevus (benign) or melanoma (malignant). This is comparable to a realistic, remote store-and-forward teledermatology task, with a more natural two- alternative forced choice (yes / no) design.10016] The study found a statistically significant serial dependence in discrimination judgments that was tuned to the sequential similarity in the malignancy of the lesions. The consequence of the serial dependence was a statistically significant reduction in metrics of sensitivity and specificity, including reduced d-prime (d’) and increased error rates.Additionally, using a recent Learned Perceptual Image Patch Similarity (LPIPS) computer vision model, the study quantified serial dependence as a function of the semantic similarity between sequential images and found that serial dependence varied as a function of the pairwise similarity between sequential images.
[0017] Overall summary statistics revealed that observers were highly sensitive to the malignancy discrimination task. Across the user population, sensitivity was 78.72%, specificity was 74.74%, d' was 1.46, c was -0.065, and the error rate was 18.6%. These metrics indicate that observers were able to perform the dermatological judgment task, consistent with the observers having some degree of expertise.
[0018] The embodiments here mitigate some of the systematic errors that occur in medical imaging, including telemedicine-based diagnosis, by decreasing the sequential biases and other observer-specific idiosyncratic errors in perceptual judgments. One particular type ofdecision bias in medical image perception judgments reported in literature is called serial dependence. Serial dependence is a sequential bias in judgments such that current reports are pulled by previous judgments and experiences. The serial dependence effect is proportional to the similarity between images and is strongest when the images are moderately similar.
[0019] Serial dependency represents one type of sequential bias, in which a current thing is judged more similar to what was just seen in the last few moments. As an example, consider stunt doubles that look more like the actors they replace, an illusion largely caused by serial dependence, which prevents audiences from noticing the changed identity. Yet, when stunt doubles stand side-by-side with the actor they replaced, they are clearly different people.Other types of sequential biases include the “gambler’s fallacy,” a bias that causes people to misjudge actually random events as more or less likely depending upon what came before, and “anchoring,” in which a person judges the current thing to the first thing the person saw that day / session / trial. Other temporal order biases include primacy and recency effects, and biases to alternate judgements.
[0020] The embodiments here include a method that can estimate pairwise image similarity via a deep learning model, the pairwise similarity indicating a probability of sequential bias.The method then reorganizes the image sequence order to minimize sequential bias. The result is a significant improvement in the accuracy of medical image judgments, such as judgments of malignancy in radiographs, photographs, slides, and other medical images. The process compares all pairs of images in the sequence to each other and then determines an order of images that maximizes the difference in similarity between all sequential pairs of images.
[0021] FIG. 1 shows an embodiment of an observation bias mitigation system to reduce observation bias in medical images sent to a remote diagnosis provider. A patient 10 undergoes some type of medical imaging process, such as a CT scan, MRI, PET scan, X-ray,mammogram, etc., using a medical imaging system 12 in a medical environment such as a clinic. The clinic then sends the images to a remote diagnosis provider. Sending the images to the remote diagnosis provider may involve the images being transferred to a computer local to the imaging system, either directly connected or through a local area network (LAN), that connects to the remote diagnosis provider, or the imaging system itself may be networked and can transmit the images directly. No limitation to any particular type of transmission between the imaging system 12 and the remote diagnosis provider is intended, and none should be inferred.
[0022] In one embodiment, the remote diagnosis provider, such as a store-and-forward telemedicine provider, may comprise one or more servers to which the clinic at which the imaging system 12 resides across a network 16. One or more computing devices, such as servers or other computing devices 18, may receive the images. The server 18 may access the machine learning system 20 through an application programming interface (API). For ease of understanding, the discussion here will refer to the machine learning system as a machine learning network. In one embodiment, the machine learning system comprises a deep learning neural network, although the machine learning system may comprise other types of neural networks, or other types of machine learning systems. No limitation to any type of machine learning model or system is intended and none should be inferred. The machine learning network may reside on the server or on another computing device, either local to the telehealth provider, or remote from it.
[0023] The images in the batch will typically have a sequence, the sequence in which the imaging system acquired the images or an unprincipled random sequence. The sequence may consist of the sequence in which the images are acquired, but may include a range of timestamps, sizes, dates, names, initials, patient IDs, domains or modalities, imaging approaches, or some proprietary label that then gets alphabetized, intentionally randomized,or some other "unprincipled" order. Generally, the sequence includes any possible sequential organization that might naturally or intentionally occur. As discussed above, sequential bias in user decisions may arise in the batch, which could give rise to judgment errors. One such judgment error can cause a significant number of misdiagnoses, such as classifying a lesion as malignant when it is benign or classifying it as benign when it is malignant.
[0024] In an embodiment, the method estimates the pairwise similarity among the batch of images, higher degrees of similarity indicate a higher probability of sequential bias. Pairwise similarity of the images in the first order may involve comparing image 1 to image 2, then image 2 to image 3, until all images have been compared and the comparison process has made a determination of how similar the images in each pair are compared to each other. This may be expressed as a probability.
[0025] The deep learning network 22 analyzes the images to estimate similarity and makes a prediction about likely sequential biases that may arise in user decisions. While the embodiments here discuss machine learning, the system may employ other approaches for measuring and quantifying similarity. The process then reorganizes the images in a sequence in which the sequential bias has been minimized to alleviate sequential effects in human judgment. This process involves ordering the images into an order that maximizes the differences between all sequential pairs of images. The output images 24 have a second, reordered sequence designed to mitigate observation bias. Provider 28 who will view the images to make a diagnosis will receive the images. The provider will then communicate the diagnosis back to the clinic so the patient can receive the diagnosis.
[0026] In the embodiment shown in FIG. 1, the bias mitigation system will be embedded in existing store-and-forward medical systems, such as the telemedicine pipeline, as a preprocessing module. A possible variation could pack the embodiments into a product and use it along with existing store -and-forward telemedicine products. In particular, theembodiments here could be inserted into existing software pipelines used by health care facilities, telehealth companies, insurance companies, hospital systems, individual clinician practices, medical software companies, or it could be incorporated into existing hardware by manufacturers.
[0027] In other embodiments, the imaging system may reside in a facility that has a local version of the bias mitigation system, such as a hospital, larger clinic, or a laboratory, such as those to which sometimes patients are sent from their doctor. The bias mitigation system resides locally to the imaging system. FIG. 2 shows such an embodiment.
[0028] In FIG. 2, the imaging machine 12 generates image 14 of patient 10. As in the remote environment, the images have a first sequence. The imaging system 12, or a connected computing device then sends images 14 to the bias mitigation system through a local area network (LAN) 26. As in the remote embodiment, the bias mitigation system may reside on the server or other computing device 18. Computing device 18, if it does not contain the bias mitigation system, then sends the images to the bias mitigation system 22 through the API 20.The bias mitigation system then operates as before to determine a level of similarity between adjacent images in the sequence and reorders them so that sequential images in the sequency have the maximized differences. The reordered sequence is then transmitted to provider 28, which may be the individual who performed the imaging, or may be elsewhere in the facility.
[0029] While FIGs. 1 and 2 show particular embodiments of the bias mitigation system, one should note that the bias mitigation system may operate in other environments. For example, as computing and imaging capabilities increase, the clinic in the embodiment of FIG. 1 may be replaced with a mobile imaging system, such as one in a vehicle such as an ambulance, or other mobile medical imaging units, including but not limited to, mobile MRIs, and mobile sonograms, as examples. The embodiment of FIG. 2 may involve providers locatedsomewhere not in the same physical building as the imaging machine, but connected to the same LAN, etc.
[0030] FIG. 3 shows a more detailed view of the server or other connected computing device18 that may perform the re sequencing attached to, or as part of, the imaging system 12. The one or more processors, represented by processor 180, may execute code, which may include the algorithm, also referred to as a model, of the deep learning network or other machine learning model. The code, and the images, may reside in one or more memories represented by memory 184. If device 18 is contained in a local environment similar to the embodiment of FIG. 3, the device 18 may include the imaging capability to generate the images, such as an X-ray machine, CT scanner, etc., or may connect directly to the imaging system 12.
[0031] FIG. 4 shows a flowchart of a general embodiment of this process that applies to the embodiments and its variations and modifications. The processor receives a batch of images in a first sequence at 30. The machine learning, or other system or process, analyzes the image similarity in the sequences at 32. The batch reorganizes the images into a second sequence at 34 in accordance with the pairwise analysis discussed above. The system then sends the batch to the provider in a reordered sequence that maximizes the differences between sequential images.
[0032] The process does not replace the human clinician. Rather, the process estimates image similarity, makes a prediction about sequential biases that will arise in humans. The process then reorganizes the image sequence before reaching the end user, in this case the clinician or provider. The resulting image sequence reduces sequential bias and increases diagnostic accuracy, increased sensitivity, increased selectivity, reduced error rate, and reduced bias.
[0033] The embodiments have several advantages. First, no mitigation strategies exist for sequential errors in medical diagnosis or remote-store-and-forward telemedicine, for any type of medicine, which suffer from sequential bias issues in all types of imaging and medicaldecisions based upon that imaging. The embodiments here directly target sequential biases in judgment including serial dependence and other temporal order decision errors, to mitigate systematic errors that have been well studied. The embodiments can be transparent to the end user or clinician. The process does not interfere with clinician practice, it does not require clinician input, and it would be seamless from the perspective of clinicians. The embodiments can be easily incorporated into existing pipelines at many possible stages, from back-end hardware to preprocessing and batching software, to front-end user interface software. The embodiments ameliorate sequential biases in judgments. One should note that it may not solve all sources of error in all medical image perception tasks.
[0034] Additionally, this written description makes reference to particular features. It is to be understood that the disclosure in this specification includes all possible combinations of those particular features. For example, where a particular feature is disclosed in the context of a particular aspect, that feature can also be used, to the extent possible, in the context of other aspects.
[0035] Also, when reference is made in this application to a method having two or more defined steps or operations, the defined steps or operations can be carried out in any order or simultaneously, unless the context excludes those possibilities.
[0036] All features disclosed in the specification, including the claims, abstract, and drawings, and all the steps in any method or process disclosed, may be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive. Each feature disclosed in the specification, including the claims, abstract, and drawings, can be replaced by alternative features serving the same, equivalent, or similar purpose, unless expressly stated otherwise.
[0037] Although specific aspects of this disclosure have been illustrated and described for purposes of illustration, it will be understood that various modifications may be made withoutdeparting from the spirit and scope of the invention. Accordingly, the invention should not be limited except as by the appended claims.
Claims
WHAT IS CLAIMED IS:
1. A system, comprising: an imaging system to capture a batch of medical images in a first sequence; and one or more processors configured to execute code the causes the one or more processors to: use a machine learning system to analyze the batch of medical images in the first sequence to determine similarity between sequential images in the batch and to determine a sequential bias; reorganizing the batch of medical images into a second, reordered batch of medical images having a sequence with a lower sequential bias than the first sequence; and transmit the reordered batch of medical images to a provider.
2. The system as claimed in claim 1, wherein the imaging system comprises one of anX-ray machine, a CT scanner, a PET scanner, a mammogram machine, a magnetic resonance imaging (MRI) machine, or a camera.
3. The system as claimed in claim 1, wherein the machine learning system resides remotely from the imaging system and the code that causes the one or more processors to transmit the reordered batch of medical images comprises code that causes the server to transmit the reordered batch to the provider.
4. The system as claimed in claim 1, wherein the machine learning system resides local to the imaging system, and the code that causes the one or more processors to transmit the reordered batch causes the one or more processors to display the reordered batch to the provider.
5. The system as claimed in claim 1, wherein the machine learning system comprises a deep learning neural network.
6. The system as claimed in claim 1, wherein the code that causes the one or more processors to use the machine learning system comprises code that causes the one or more processors to access the machine learning system through an application programming interface.
7. The system as claimed in claim 1 , wherein the code that causes the one or more processors to use a machine learning system to determine similarity between sequential images comprises code to cause the one or more processors to use the machine learning system to perform a pairwise analysis of pairs of images in the first sequence.
8. The system as claimed in claim 1, wherein the code that causes the one or more processors to reorganize the batch of medical images comprises code causes the one or more processors to reorganize the batch of medical images into the reordered batch of medical images to maximize differences between the images in the batch of medical images.
9. A method, comprising: receiving a batch of medical images in a first sequence; using a machine learning system to analyze the batch of medical images to determine similarity between sequential images in the batch based upon probabilities of a sequential bias; reorganizing the batch of medical images into a second, reordered batch of medical images with a lower sequential bias probability than the first sequence; and transmitting the batch of medical images in the second sequence to a provider.
10. The method as claimed in claim 9, receiving the batch of medical image comprises receiving the batch of medical images from one of an X-ray machine, a CT scanner, a PET scanner, a mammogram machine, a magnetic resonance imaging (MRI) machine, and a camera.
11. The method as claimed in claim 9, wherein the machine learning system resides in a server separate from the imaging system and transmitting the reordered batch of medical images comprises transmitting the reordered batch across a network to the provider.
12. The method as claimed in claim 9, wherein the machine learning system resides in the imaging system, and the transmitting of the reordered batch of images comprises displaying the reordered batch to the provider.
13. The method as claimed in claim 9, wherein the machine learning system comprises a deep learning neural network.
14. The method as claimed in claim 9, wherein using the machine learning system analyze the batch of medical images to determine similarity between sequential images comprises using the machine learning system to perform a pairwise analysis of pairs of images in the first sequence.
15. The method as claimed in claim 9, wherein reorganizing the batch of images into a second, reordered batch of medical images comprises reordering the batch of images to maximize differences between sequential image pairs.
Citation Information
Patent Citations
Biomarker panels, diagnostic methods and test kits for ovarian cancer
US20150031561A1
Methods for transferring a plurality of medical images
US20220344031A1
Computer implemented method for automated analysis of the bias in measurements performed on medical images of an anatomical structure
US20220392648A1
