System and method for detecting anomalies in pet radiology images

Abstract

In one embodiment, a method includes accessing radiographic images of an animal, where one or more first radiographic images of the radiographic images depict the animal from one or more perspectives, respectively, and wherein one or more second ones of the radiographic images depict one or more body parts of the animal, respectively, determining a disease classification associated with the animal based on analyzing the radiographic images by a machine learning model, generating a diagnostic report associated with the animal based on the machine learning model, wherein the diagnostic report includes a disease classification and a natural language text radiology report, and sending an instruction to the user device for presenting the diagnostic report.

Description

[0001] priority

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 358,905, filed on July 7, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to using one or more machine learning models or tools to evaluate pet or animal radiology images. Background Art

[0004] Veterinarians are increasingly utilizing image-based diagnostic techniques, such as X-rays, to diagnose or identify health problems in animals or pets. However, there are fewer than 1,100 radiologists with veterinary training worldwide. As a result, many veterinarians are unable to take advantage of the benefits offered by image-based diagnostic techniques. Even for those veterinarians with radiology training, reviewing medical images can be time-consuming and cumbersome. Exacerbating these difficulties is that animal or pet radiology images may be misoriented and / or missing or have incorrect lateral markings. Therefore, there is a need for a system that can automatically process and interpret pet diagnostic images and return clinically reliable results to veterinarians with or without radiology training. Summary of the invention

[0005] In certain non-limiting embodiments, the present disclosure provides systems and methods for training and using machine learning models to process, interpret and / or analyze radiological images of animals or pets. The images can be in any format used for diagnosis of medical conditions, such as Digital Imaging and Communications in Medicine (DICOM), as well as other formats for displaying images. In a specific embodiment, the radiological image can be associated with a radiology report. Typically, for radiological image analysis, text-specific (e.g., natural language processing) models and image-specific models can be trained based on radiological reports and radiological images, respectively. When deployed, text-specific models and image-specific models can also be deployed as independent entities and (in principle) can be used separately from each other. Compared to these traditional methods, the embodiments disclosed herein can train a text-image joint model that can be used to directly generate and / or verify radiological reports based on radiological images. In one embodiment, the text-image joint model can be programmed to detect abnormalities based on animal or pet radiological images.

[0006] In one embodiment, the present disclosure provides a system and method for automatically detecting abnormalities based on animal or pet radiographic images. In various embodiments, one or more machine learning models or tools can be used to analyze and / or detect abnormalities in acquired, collected and / or received images. In some embodiments, the machine learning model may include one or more neural networks. As an example and not limitation, the neural network may be a convolutional neural network (CNN). In one embodiment, abnormality detection may, for example, indicate healthy tissue or abnormal tissue. In one embodiment, tissue classified as abnormal may be further classified as, for example, cardiovascular, pulmonary structure, mediastinal structure, pleural cavity and / or extrathoracic.

[0007] In some embodiments, the present disclosure provides a method for performing abnormality detection based on radiographic images of an animal or pet by one or more computing systems. The method includes: accessing multiple radiographic images of the animal, wherein one or more first radiographic images in the multiple radiographic images depict the animal from one or more perspectives, respectively, and wherein one or more second radiographic images in the multiple radiographic images depict one or more body parts of the animal, respectively; determining one or more disease classifications associated with the animal based on an analysis of the multiple radiographic images by a machine learning model; generating a diagnostic report associated with the animal based on the machine learning model, wherein the diagnostic report includes one or more disease classifications and a natural language text radiology report; and sending an instruction to a user device for presenting the diagnostic report.

[0008] In one embodiment, each of the plurality of radiographic images is formatted as a Digital Imaging and Communications in Medicine (DICOM) image.

[0009] In one embodiment, the machine learning model is based on at least one first neural network and at least one second neural network, and the at least one first neural network and the at least one second neural network are coupled to each other.

[0010] In one embodiment, generating a diagnosis report includes: accessing a plurality of reference reports; encoding the plurality of reference reports into a feature space; encoding a plurality of radiographic images into the feature space; and determining the diagnosis report based on a similarity search in the feature space.

[0011] In one embodiment, a disease classification of the one or more disease classifications indicates abnormal tissue.

[0012] In one embodiment, the method further comprises identifying the abnormal tissue as at least one of a cardiovascular, a pulmonary structure, a mediastinal structure, a pleural cavity, or an extra-thoracic cavity.

[0013] In one embodiment, the method further includes: accessing a plurality of training radiological image images, wherein the plurality of training radiological image images are respectively associated with a plurality of training radiology reports; and training a machine learning model based on the accessed training radiological image images and their respective training radiology reports.

[0014] In one embodiment, the method further comprises: preprocessing each of the plurality of training radiological image images, wherein the preprocessing comprises one or more of padding, random enhancement, random flipping, Gaussian blurring or normalization.

[0015] In one embodiment, the method further comprises applying a long document encoding to each of the plurality of training radiology reports.

[0016] In one embodiment, the method further comprises: preprocessing each of the plurality of training radiology reports, wherein the preprocessing comprises one or more of tokenizing, padding, adding classification tags, or applying an attention mask.

[0017] In one embodiment, the machine learning model includes an image encoder, a multi-image encoder, a text decoder, and a multimodal decoder.

[0018] In one embodiment, the method also includes: generating a feature map based on multiple radiological images by an image encoder; generating one or more multi-image keys and values ​​based on the feature map by a multi-image encoder; and generating a natural language text radiology report based on the one or more multi-image keys and values ​​and a sentence start marker by a multimodal decoder.

[0019] In one embodiment, the diagnostic report further includes one or more radiological images from among the plurality of radiological images.

[0020] In various embodiments, the present disclosure provides one or more computer-readable non-transitory storage media that, when executed by one or more processors, are operable to perform one or more methods provided by the present disclosure.

[0021] In various embodiments, the present disclosure provides a system comprising: one or more processors; and one or more computer-readable non-transitory storage media, which are connected to the one or more processors and contain instructions, which, when executed by the one or more processors, are operable to cause the system to perform one or more methods provided by the present disclosure.

[0022] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited thereto. Certain non-limiting embodiments may include all, some or none of the components, elements, features, functions, operations or steps of the embodiments disclosed herein. Embodiments according to the present invention are disclosed in particular in the appended claims relating to methods. The dependencies or references in the appended claims are selected only for formal reasons. However, any subject matter resulting from intentional reference to any previous claim (especially multiple dependencies) may also be claimed, and therefore, no matter which dependencies are selected in the appended claims, any combination of claims and their features may be disclosed and claimed. The subject matter that may be claimed includes not only the combination of features listed in the appended claims, but also any other combination of features in the claims, wherein the individual features mentioned in the claims may be combined with any other features or combinations of other features in the claims. In addition, any embodiment and feature described or depicted herein may be claimed in a separate claim, and / or in any combination with any embodiment or feature described or depicted herein, or with any feature of the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 An example architecture of a comparative radiology captioning model is shown.

[0024] Figure 2 An example method for abnormality detection in radiographic images of animals is shown.

[0025] Figure 3 An example computer system or apparatus for facilitating abnormality detection from radiographic images of an animal is shown. DETAILED DESCRIPTION

[0026] The terms used in this specification generally have their ordinary meanings in the art in the context of the present disclosure and in the specific context in which each term is used. Certain terms are discussed below or elsewhere in the specification to provide additional guidance in describing the compositions and methods of the present disclosure and how to make and use them.

[0027] As used in the specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0028] As used herein, the terms “comprises,” “comprising,” or any other variations thereof, are intended to cover a non-exclusive inclusion such that a process, method, article, system, or apparatus that comprises a list of elements includes not only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0029] As used herein, the term "animal" or "pet" used in accordance with the present disclosure refers to domestic animals, including but not limited to domestic dogs, domestic cats, horses, cattle, ferrets, rabbits, pigs, rats, mice, gerbils, hamsters, goats, etc. Domestic dogs and domestic cats are specific non-limiting examples of pets. The term "animal" or "pet" used in accordance with the present disclosure may also refer to wild animals, including but not limited to bison, elk, deer, wild deer, ducks, poultry, fish, etc.

[0030] As used herein, a "feature" of an image or slice can be determined based on one or more measurable characteristics of the image or slice. For example, a feature can be a blemish in the image, a dark spot, or tissue with different sizes, shapes, or light intensity levels.

[0031] In the detailed description herein, references to "an embodiment", "an embodiment", "an embodiment", "in various embodiments", "certain embodiments", "some embodiments", "other embodiments", "certain other embodiments", etc., indicate that the embodiment may include specific features, structures, or characteristics, but every embodiment may not necessarily include the specific features, structures, or characteristics. In addition, such phrases do not necessarily refer to the same embodiment. In addition, when specific features, structures, or characteristics are described in conjunction with an embodiment, whether or not explicitly described, it is considered to be within the knowledge of those skilled in the art to affect such features, structures, or characteristics in conjunction with other embodiments. After reading the specification, it will be clear to those skilled in the relevant art how to implement the present disclosure in alternative embodiments.

[0032] As used herein, the term "device" refers to a computing system or mobile device. For example, the term "device" may include a smartphone, a tablet computer, or a laptop computer. In particular, the computing system may include functionality for determining its position, direction, or orientation, such as a GPS receiver, a compass, a gyroscope, or an accelerometer. The client device may further include functionality for wireless communications, such as Bluetooth communications, near-field communications (NFC) or infrared (IR) communications, or communications with a wireless local area network (WLAN) or a cellular telephone network. Such a device may also include one or more cameras, scanners, touch screens, microphones, or speakers. The client device may also execute software applications, such as games, web browsers, or social networking applications. The client device may, for example, include a user device, a smartphone, a tablet computer, a laptop computer, a desktop computer, or a smart watch.

[0033] The exemplary processes and embodiments may be operated or performed by a computing system or client device through a mobile application and an associated graphical user interface ("UX" or "GUI"). In certain non-limiting embodiments, the computing system or client device may be, for example, a mobile computing system, such as a smartphone, a tablet, or a laptop. The mobile computing system may include functionality for determining its position, direction, or orientation, such as a GPS receiver, a compass, a gyroscope, or an accelerometer. Such a device may also include functionality for wireless communications, such as Bluetooth communications, near field communications (NFC), or infrared (IR) communications, or communications with a wireless local area network (WLAN), 3G, 4G, LTE, LTE-A, 5G, the Internet of Things, or a cellular telephone network. Such a device may also include one or more cameras, scanners, touch screens, microphones, or speakers. The mobile computing system may also execute software applications, such as games, web browsers, or social networking applications. Through social networking applications, users can connect, communicate, and share information with other users in their social network.

[0034] The terms used in this specification generally have their ordinary meanings in the art in the context of the present disclosure and in the specific context in which each term is used. Certain terms are discussed below or elsewhere in the specification to provide additional guidance in describing the compositions and methods of the present disclosure and how to make and use them.

[0035] In recent years, semi-supervised multimodal artificial intelligence (AI) models have achieved state-of-the-art results in various downstream tasks. The embodiments disclosed herein utilize the effectiveness of these methods for disease classification and report generation in the field of veterinary radiology. Specifically, a comparative radiology annotation model is disclosed herein. The architecture of the comparative radiology annotation model can align X-ray images using contrast and annotation losses and report at the global and local levels. Because multiple different views and body parts are used to write diagnostic reports, the architecture can align multiple X-ray images into a single report. Experimental results show that the architecture can significantly improve the performance of multiple radiological examination results compared to supervised training methods using alternative labeling methods. Ablation studies were also performed to demonstrate the importance of each architectural design choice. The text generation capabilities of the comparative radiology annotation model highlight the potential of using multimodal large language models to generate radiology reports. The comparative radiology annotation model can be a powerful architecture for training large unlabeled datasets using multi-image text pair inputs.

[0036] Artificial intelligence systems using supervised learning methods can be used to assist veterinary radiologists in interpreting X-ray images. These methods may rely on manually labeling X-ray images for disease classification, which is a time-consuming and resource-intensive process. In recent years, semi-supervised multimodal methods have achieved great success in achieving state-of-the-art performance on a variety of downstream tasks. These methods can leverage readily available text as ground truth labels, thereby reducing the need for labeled data. Dataset size and model performance tend to be positively correlated. Therefore, the use of semi-supervised methods can improve model performance by training with large unlabeled datasets. This development is of great significance to the field of radiology, as models can now be trained using large amounts of historical reports that are routinely generated along with X-ray images.

[0037] In addition, these state-of-the-art models have demonstrated the benefits of multimodal approaches to unimodal model performance. Contrasting methods can achieve zero-shot capabilities by learning an image-text joint embedding space to align similar text and images. In addition, optimizing the generative loss of cross-modal alignment has been shown to improve the model's ability to learn fine-grained local feature representations. Therefore, the embodiments disclosed herein disclose a method for training radiological image-text pairs for disease classification and text generation using contrast and generative methods. In certain non-limiting embodiments, the present disclosure provides automated techniques for detecting abnormalities based on animal or pet radiographic images. One or more radiographic images may be in digital imaging and communications in medicine (DICOM) format. Once the image is received, it can be analyzed using a trained machine learning model or tool (e.g., a neural network model) to determine abnormalities based on these radiographic images. In some embodiments, the method may use a visual encoder and a decoupled text unimodal and multimodal decoder method.

[0038] However, a particular challenge in developing models in the field of radiology can be that reports for a single patient often involve multiple images. This may be because multiple X-ray images are often taken during a patient visit, for example of different body parts and views. Recent work has highlighted the importance of cross-modal alignment of relevant images with reports. The embodiments disclosed herein demonstrate that incorporating images from previous patient visits can improve model performance by reducing ambiguity in reports caused by missing contextual information in the images. Prior work has also demonstrated that considering multiple image views using a CNN-ViT architecture can improve performance on multi-label classification tasks in radiology. Therefore, the method disclosed herein similarly uses a hybrid CNN-Transformer architecture as a visual encoder to facilitate multi-image embedding.

[0039] Some of the example routine work related to the embodiments disclosed herein may include RapidRead and StudyFormer. RapidRead is a deployed AI veterinary radiology system. The system can use a set of CNN models for disease classification and an expert system for evaluation generation. The methods in the present disclosure can be compared with models from this system. The StudyFormer model can generate study-level embeddings for patients using a single-image CNN encoder model and a multi-image ViT encoder model. The architecture of the comparative radiology annotation model can use the StudyFormer architecture as the visual encoder with some structural changes.

[0040] Embodiments disclosed herein disclose a contrastive radiology annotation model that can be based on a self-supervised framework for language processing in radiology field vision. In certain non-limiting embodiments, the contrastive radiology annotation model can be based on one or more neural networks. As an example and not limitation, the neural network can be based on a convolutional neural network, a transformer-based network, or an MLP mixer. In some embodiments, the architecture of the contrastive radiology annotation model can include a hybrid CNN-ViT visual encoder, a text decoder, and a multimodal decoder.

[0041] In certain non-limiting embodiments, the comparative radiology annotation model can be trained by jointly training at least two coupled neural networks, wherein at least one first network is used for radiology images and at least one second network is used for radiology reports. The at least one first network for radiology images can be considered as an image encoder, and the at least one second network for radiology reports can be considered as a text encoder. The network can be based on any suitable architecture, such as Resnet50.

[0042] In some non-limiting embodiments, the joint training of the first network and the second network can be based on multiple pairs of radiographic images and radiology reports. The comparative radiology annotation model can be trained to predict the correct pairing of radiographic images and radiology reports in the training examples. In some embodiments, the training can include learning a multimodal embedding space by jointly training the first network and the second network to maximize the cosine similarity of correctly paired radiographic image embeddings and radiology report embeddings while minimizing the cosine similarity of incorrectly paired embeddings.

[0043] While in some examples, neural networks can train learned weights for each input-output pair, CNNs can convolve trainable fixed-length kernels or filters along their inputs. In other words, CNNs can learn to recognize small, primitive features (low-level) and combine them in complex ways (high-level). In certain embodiments, CNNs can be supervised, semi-supervised, or unsupervised.

[0044] In certain non-limiting embodiments, pooling, padding, and / or striding can be used to reduce the output size of a CNN in the convolutional execution dimension, thereby reducing computational cost and / or reducing the likelihood of overtraining. Striding can describe the step size or number of steps that the filter window slides, and padding can include padding certain areas of the data with zeros before or after striding to buffer the data. In one embodiment, pooling can include, for example, simplifying the information collected by a convolutional layer or any other layer and creating a compressed version of the information contained within the layer.

[0045] In some examples, a region-based CNN (RCNN) or a one-dimensional (1-D) CNN can be used. RCNN includes using selective search to identify one or more regions of interest in an image, and independently extracting CNN features from each region for anomaly detection. The type of RCNN used in one or more embodiments may include fast RCNN, faster RCNN, or mask RCNN. In other examples, a 1-D CNN can process a fixed-length time series segment generated with a sliding window. This 1-D CNN can run in a many-to-one configuration that uses pooling and striding to connect the output of the final CNN layer. Then, a fully connected layer can be used to detect on one or more time steps.

[0046] In some embodiments, one or more CNN models and one or more LSTM models can be combined. The combined model can include a stack of four non-stride CNN layers, which can be followed by two LSTM layers and a softmax classifier. The Softmax classifier can normalize the probability distribution, which includes multiple probabilities proportional to the exponentials of the input. For example, the input signal of the CNN is not padded, so although the layers are not strided, each CNN layer shortens the time series by several samples. The LSTM layer is unidirectional, so the softmax classification corresponding to the final LSTM output can be used for training and evaluation, as well as reassembling the output time series from the sliding window fragments. But the combined model can run in a many-to-one configuration.

[0047] Figure 1 An example architecture 100 of a comparative radiology annotation model is shown. In certain non-limiting embodiments, the comparative radiology annotation model can be trained using multiple radiological images and their associated radiology reports (i.e., multiple image / text pairs 110). The multiple image / text pairs 110 can include study images 112 and their corresponding radiology reports 114. As an example and not limitation, the radiological images can be radiographs, CT scans, etc. In contrast to labels or tags, radiology reports can contain long, unstructured text descriptions of abnormalities. For example, a radiology report can contain a description that "this dog has a large heart," which corresponds to a label / tag of "cardiac hypertrophy." Therefore, training a machine learning model based on radiology reports containing long, unstructured text descriptions is more challenging than traditional training based on labels or tags.

[0048] In certain non-limiting embodiments, the visual encoder can be a hybrid CNN-b transformer based on the StudyFormer architecture. The CNN architecture used can be an Efficient-Net model pre-trained using multi-label classification on single-view X-ray images. Each image in the study is first passed through the CNN image encoder 120 individually to obtain a feature map of dimension (2048×10×10). The feature maps of all images in the study can then be concatenated to form a feature map 122 of dimension (2048×50×10). The feature map 122 can then pass through the ViT multi-image encoder 130, which outputs a vector representation of all images in the study with a dimension of (501×768). The vector representation may include a CLS embedding 140a. In some embodiments, the ViT multi-image encoder 130 may be based on a patch size of 1, a depth of 12, an attention head of 12, a multi-layer perceptron (MLP) dimension of 2048, and an output dimension of 500×768.

[0049] In some embodiments, both the unimodal text decoder and the multimodal text decoder can be small pre-trained Generative Pretrained Transformer 2 (GPT2) models. The output dimension of the embedded unimodal text decoder 150 can be [513, 768]. The output can include CLS embedding 140b. The CLS embedding 140 from the unimodal model can be used to calculate the contrast loss.

[0050] The output of the unimodal text decoder 150 can be used as a text query 152 in the multimodal text decoder 160 for a cross-attention mechanism. The output of the ViT multi-image encoder 130 can be used as a multi-image key and value 132. One output 162 of the multimodal text decoder 160 can include a probability distribution of each position in the sequence over the GPT2 corpus. Another output 164 of the multimodal text decoder 160 can include an annotation loss, which can be calculated using a tokenized ground-truth text label and a predicted text. The output dimension can be [50257, 512].

[0051] In some embodiments, both GPT2 models can be trained using low-rank adaptation of a large language model. Low-rank adaptation of a large language model can use low-rank decomposition to learn low-rank matrices in the attention layer. These matrices can represent the change from the original GPT2 weights to the weights of the new task. As an example and not limitation, low-rank adaptation of a large language model with a rank of 8 is used in this disclosure.

[0052] In certain non-limiting embodiments, radiological images and radiology reports may need to be preprocessed before training the comparative radiology annotation model. For example, the size of the radiological images can be large, such as up to 456×456 pixels. The size of all images can be adjusted to 300×300. The maximum number of images for each study can be limited to 5. In one embodiment, image cropping can be avoided by padding the radiological images. As an example and not limitation, studies with less than 5 images can be padded to meet the shape requirement of [5, 3, 300, 300]. Transformations applied to training images may include square padding, random augmentation, random flipping, and Gaussian blurring. For all images, normalization of [0.5, 0.5, 0.5] for the mean and standard deviation can be applied.

[0053] In some embodiments, preprocessing of radiology reports can be based on long document encodings rather than the commonly used short document encodings. In an alternative embodiment, a model can be trained to preprocess radiology reports based on research notes. As an example and not limitation, radiology reports can be processed using the GPT2 tagger with a maximum token length of 512. To ensure consistency, texts with fewer than 512 tokens can be padded with End-of-Sequence (EOS) markers. In addition, a Classification Token (CLS) can be added to each tokenized text to facilitate contrastive learning. An attention mask can be applied to the padded tokens.

[0054] Table 1 lists the definitions of symbols used in the formulas in the embodiments disclosed herein.

[0055]

[0056] Table 1: Definition of symbols used in the formulas

[0057] In certain non-limiting embodiments, the contrastive loss can be used to learn discriminative features by forcing the model to minimize the distance between similar instances and maximize the distance between dissimilar instances in the embedding space.

[0058] The annotation loss can optimize the model so that it learns to generate reports describing the input image. Given an image I and its corresponding ground truth annotation C = c1 c2 ... c T , where T is the length of the annotation, the annotation loss function can be defined as the negative log-likelihood of the correct word sequence:

[0059]

[0060] Here, p(c t |I,c 1:t＝1) Indicates that given an image I and the previous word c1:t＝1 In the case of generating the correct word c at time step t t probability.

[0061] Combined loss L 组合 The definition is as follows:

[0062] First, we can calculate the similarity matrix S, where each element Sij is the dot product of ri and ij, scaled by the temperature parameter θ:

[0063] S ij = <r i ,i j >·e θ ,i,j=1,2,…,n

[0064] Second, we can use the label L as the target and convert the contrast loss L 对比 Calculated as the similarity matrix S and its transpose S T The average of the cross entropy losses calculated above:

[0065]

[0066] Third, the annotation loss L can be 对比 Calculated as in the forecast report r 预测 and true report 真实 The cross entropy loss calculated above is:

[0067] L 标注 =F CE (r 预测 ,T 真实 )

[0068] Finally, the combined loss can be the sum of the contrast loss and the annotation loss, each loss weighted by its respective weight w c and w cap Zoom:

[0069]

[0070] In some embodiments, the training data consists of 3,200,173 image-text pairs. It consists of 755,263 studies. The validation dataset consists of 50,000 image-text pairs, consisting of 10,446 studies. The radiology reports consist of findings and assessments. In the embodiments disclosed herein, the model is trained only on the findings.

[0071] In some embodiments, the contrast radiology annotation model is further fine-tuned on the fine-tuning dataset. The training data of the fine-tuning dataset consists of 594,449 images and labels. It consists of 145,486 studies. The validation dataset consists of 10,005 images and labels, consisting of 2,253 studies. Each image has 41 corresponding labels indicating whether a certain finding is present in the image. Each label is for a single X-ray view. Therefore, the label of a study is determined by finding the maximum value of each label across all images in the study. Therefore, if the finding is present in any image in the study, the patient is considered to have pathology.

[0072] To validate the model performance, the visual encoder was fine-tuned on a disease classification task. This was achieved by adding a classification layer with 41 outputs to the visual encoder. The model was trained using binary cross entropy loss. Several experiments were performed using this approach, including ablation studies, to understand the impact of each part of the contrast radiology annotation model design on the classification performance.

[0073] The Contrastive Radiology Annotation model was trained for 2 weeks on a single GPU. The training lasted for 11 training epochs and was stopped when the validation loss performance plateaued. Fine-tuning the StudyFormer visual encoder using the Contrastive Radiology Annotation model weights took 5 days on a single GPU. This was also stopped when the mean precision score stopped increasing. This took 50 training epochs.

[0074] In certain non-limiting embodiments, the contrastive radiology annotation model can be used to detect abnormalities in any new radiological image. In addition, the contrastive radiology annotation model can generate a diagnostic report for one or more input radiological images, rather than just predicting labels or tags for such images. For example, the contrastive radiology annotation model can generate a diagnostic report that includes a text description of the image (e.g., "This dog has a large heart"), rather than predicting "cardiac hypertrophy" for one or more radiological images. In one embodiment, in order to enable the contrastive radiology annotation model to generate a diagnostic report, the following steps can be used. First, the contrastive radiology annotation model can encode a reference diagnostic report into a feature space. Then, the contrastive radiology annotation model can encode the input radiological image into the shared feature space and perform a similarity search. The contrastive radiology annotation model can also select the reference diagnostic report that is closest to the input radiological image as the output diagnostic report.

[0075] In certain non-limiting embodiments, in addition to detecting anomalies, the contrast radiology annotation model may also determine more detailed information about the detected anomalies. As an example and not limitation, after detecting an anomaly, the contrast radiology annotation model may also determine a location description associated with the anomaly, a size description associated with the anomaly, or a severity description associated with the anomaly.

[0076] In certain non-limiting embodiments, the contrast radiology annotation model can be used for various clinical or medical purposes. For example, a veterinarian or veterinary assistant can take a radiographic image of a pet. The image can then be processed using the contrast radiology annotation model. During processing, the image can be classified as normal or abnormal. If abnormal, the image can be classified as at least one of cardiovascular, pulmonary structure, mediastinal structure, pleural cavity, or extrathoracic cavity. The image can be further analyzed to determine a location description, size description, or severity description associated with the abnormality. In some non-limiting embodiments, the image can be subclassified. For example, a subclass of the pleural cavity can include pleural effusion, pneumothorax, and / or pleural mass. Similarly, the image can be further analyzed to determine a location description, size description, or severity description associated with a subclass. The image can then be displayed to the user together with the determined image abnormality category and subclass. The image can be displayed on a screen or computing device associated with the user.

[0077] In one embodiment, the comparative radiology annotation model and resulting images can be used to provide a second opinion to a radiologist on demand, forming the basis for a service that provides immediate evaluation of radiological images to veterinary hospitals, and / or improving efficiency and productivity by allowing radiologists to focus on the pet itself rather than the images.

[0078] Specific embodiments disclosed herein conducted experiments to validate the effectiveness of the comparative radiology annotation model. The comparative radiology annotation model was evaluated by comparing its performance with the current model ensemble deployed in the RapidRead system. Performance was measured using ROCAUC and average precision metrics. In summary, the comparative radiology annotation model had higher ROCAUC for 15 findings and higher average precision for 10 findings. Table 2 shows the findings that outperformed the current ensemble on at least one metric.

[0079]

[0080]

[0081] Table 2: Comparison of ROCAUC and mean average precision scores of contrasted radiology annotation models with current ensembles.

[0082] To evaluate the impact of multimodal training on the performance of StudyFormer, a StudyFormer model with Image-Net weights was also fine-tuned on the same data. The mean average precision and ROCAUC scores of this model were then compared to those of StudyFormer trained on a comparative radiology annotation model. This showed that for most findings, the performance of the two metrics differed significantly.

[0083] In another experiment, the multimodal text decoder was removed (so the contrastive radiology model had no annotator) to understand the impact of generative learning on the model's classification performance. This means that only the image encoder and text decoder weights were optimized using the contrastive loss. The results of this study showed that while most findings performed better compared to ImageNet StudyFormer, most findings performed significantly worse compared to StudyFormer trained with the contrastive radiology annotation model.

[0084] Table 3 compares the mean accuracy results of the comparative radiomics annotation model and ablation studies.

[0085]

[0086]

[0087]

[0088] Table 3: Comparison of mean accuracy scores in ablation studies.

[0089] The text generation capabilities of the comparative radiology annotation model were tested using unseen test data. First, multi-image X-ray embeddings were generated using the StudyFormer visual encoder. These were then used as keys and values ​​in the multimodal decoder. The sentence start token was used as the initial query. The multimodal decoder then autoregressively generated a text sequence using the key, value, and query, which in this case was a diagnostic report. This text was compared to human reports. The comparison showed that the model can generate accurate reports that are very similar to how humans interpret images and write reports. It also showed that the model generates a lot of information that is not present in human text. There was significant variation in how well the generated text aligned with human text across studies.

[0090] In an alternative embodiment, a pre-trained Resnet50 architecture may be used instead. Figure 1The architecture disclosed in the training is configured to detect abnormalities in a machine learning model based on an animal or pet radiographic image. The embodiments disclosed herein also experimented with a trained model based on the Resnet50 architecture. Resnet50 from OpenAIs CLIPS library (i.e., a public library) was used to generate image features for 72,105 training images and 10,477 test images. Logistic regression training was performed on the features of the training images and the radiology reports, and then tested on the features and labels of the test images. For each of the 39 labels, ROC-AUC was calculated. The average of all 39 ROC-AUCs of the trained machine learning model was 0.7761819903717145. In contrast, the average of all 39 ROC-AUCs of OpenAI Clip (i.e., the most advanced method) was 0.7613616312748511. Table 4 illustrates the comparison of ROC-AUCs of 39 labels between the trained machine learning model and OpenAI Clip. The comparison results show that the trained machine learning model improves performance over the prior art.

[0091]

[0092]

[0093]

[0094] Table 4. ROC-AUC comparison of machine learning models based on Resnet50 and OpenAI Clip.

[0095] The embodiments disclosed herein study the effectiveness of multimodal training methods on the performance of computer vision models for disease classification in feline and canine radiographs. The embodiments disclosed herein disclose a comparative radiology annotation model that utilizes a novel model architecture to train multiple image / text pairs using contrast and annotation losses. The embodiments disclosed herein demonstrate that after multimodal alignment, performance on multi-label image classification tasks is significantly better for several findings, and is otherwise comparable to the model ensemble deployed in the current RapidRead system.

[0096] Interestingly, some of the more difficult findings in current systems have the most significant performance improvements after training with this architecture. For example, the average precision score for “Gastric Intake” is 52% higher using the comparative radiology annotation model compared to the current ensemble. This may be because the current system uses a supervised training approach, where most labels are derived using an NLP algorithm. The algorithm can extract labels from the radiology report using rules. However, this multimodal approach of the comparative radiology annotation model can use the text itself as the ground truth label. This may mean that it can capture subtle differences in the text that may be missed by the NLP labeler, such as syntactic variations in pathology descriptions. Therefore, the present disclosure shows that alignment between radiology reports and X-ray images can achieve better representation learning for pathologies that are difficult to label using alternative methods.

[0097] Ablation studies were also performed to understand the importance of different parts of the architecture. The disclosure first shows that training the StudyFormer model with ImageNet weights leads to significantly worse results than training with weights trained with a comparative radiology annotation model. This suggests that performance improvements can come from multimodal training and are not simply a result of using the StudyFormer architecture itself. The disclosure then highlights the importance of the multimodal decoder by showing that removing it leads to a significant drop in performance compared to StudyFormer trained with a comparative radiology annotation model.

[0098] In the embodiments disclosed herein, the maximum number of images used per study is 5.75% of the studies have 5 or fewer images. Therefore, 5 was chosen because it captures all images in most studies while maintaining the requirement for padding images and low computational cost. However, this may mean that 25% of the studies have excess images that cannot be used. As a result, some images referenced in the study reports are not accessible to the model. This may have affected the model's ability to accurately align images and reports in the embedding space.

[0099] The embodiments disclosed herein may have an impact on the future development and deployment of deep learning models in the field of radiology. The present disclosure demonstrates that a multimodal approach can be used to train a model using multiple X-ray images and their corresponding reports. This can allow the use of large unlabeled datasets without the need for alternative labeling methods (e.g., NLP algorithms). Specifically, the present disclosure demonstrates that such training methods may be particularly beneficial in the field of radiology for findings that are difficult to reliably detect using alternative labeling methods.

[0100] The embodiments disclosed herein also highlight the potential of using large language models to automate the radiology report writing process. The present disclosure shows that by simply providing unseen X-ray images to a comparative radiology annotation model, the model can generate text that is very similar to human diagnostic reports.

[0101] In summary, the present disclosure demonstrates that the architecture of the contrastive radiology annotation model can be a powerful approach to training deep learning models compared to supervised learning with alternative labeling methods. The present disclosure also highlights the potential for automated diagnostic report generation by comparing actual reports to those generated by the contrastive radiology annotation model. Overall, the embodiments disclosed herein demonstrate the potential benefits of using the architecture of the contrastive radiology annotation model to train models for image classification tasks when using large unlabeled datasets with multiple image / text pair inputs.

[0102] Figure 2 An example method 200 for abnormality detection of radiographic images of an animal is shown. In step 210, one or more computing systems may access a plurality of radiographic images of the animal, wherein one or more first radiographic images of the plurality of radiographic images respectively depict the animal from one or more perspectives, and wherein one or more second radiographic images of the plurality of radiographic images respectively depict one or more body parts of the animal. In step 220, the computing system may analyze the plurality of radiographic images based on a machine learning model to determine one or more disease classifications associated with the animal. In step 230, the computing system may generate a diagnostic report associated with the animal based on the machine learning model, wherein the diagnostic report includes one or more disease classifications and a natural language text radiology report. In step 240, the computing system may send an instruction to a user device for presenting the diagnostic report.

[0103] Figure 3 An example computer system 300 or device for facilitating abnormality detection based on radiological images of animals is shown. In certain non-limiting embodiments, one or more computer systems 300 perform one or more steps of one or more methods described or illustrated herein. In certain other non-limiting embodiments, one or more computer systems 300 provide the functionality described or illustrated herein. In certain non-limiting embodiments, software running on one or more computer systems 300 performs one or more steps of one or more methods described or illustrated herein, or provides the functionality described or illustrated herein. Some non-limiting embodiments include one or more portions of one or more computer systems 300. Here, where appropriate, references to computer systems may include computing devices, and vice versa. In addition, where appropriate, references to computer systems may include one or more computer systems.

[0104] The present disclosure contemplates any suitable number of computer systems 300. The present disclosure contemplates computer systems 300 in any suitable physical form. As an example and not limitation, computer system 300 may be an embedded computer system, a system-on-chip (System-on-Chip, SOC), a single-board computer system (SBC) (e.g., a module computer (Computer-on-Module, COM) or a system-on-module (System-on-Module, SOM)), a desktop computer system, a notebook computer or laptop computer system, an interactive kiosk, a mainframe computer, a computer system network, a mobile phone, a personal digital assistant (Personal Digital Assistant, PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more thereof. Where appropriate, computer system 300 may include one or more computer systems 400, which are centralized or distributed, across multiple locations, across multiple machines, across multiple data centers, or exist in the cloud, where the cloud may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 300 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not limitation, one or more computer systems 300 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. Where appropriate, one or more computer systems 300 may perform one or more steps of one or more methods described or illustrated herein at different times or at different locations.

[0105] In certain non-limiting embodiments, computer system 300 includes processor 302, memory 304, storage 306, input / output (I / O) interface 308, communication interface 410, and bus 412. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0106] In some non-limiting embodiments, the processor 302 includes hardware for executing instructions (e.g., instructions that constitute a computer program). By way of example and not limitation, to execute instructions, the processor 302 may retrieve (or fetch) instructions from an internal register, an internal cache, memory 304, or storage 306; decode and execute the instructions; and then write one or more results to an internal register, an internal cache, memory 304, or storage 306. In some non-limiting embodiments, the processor 302 may include one or more internal caches for data, instructions, or addresses. The present disclosure contemplates that the processor 302 includes any suitable number of any suitable internal caches, where appropriate. By way of example and not limitation, the processor 302 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of instructions in the memory 304 or storage 306, and the instruction cache may speed up the retrieval of these instructions by the processor 302. The data in the data cache may be a copy of the data in the memory 304 or the storage 306 for operation by the instructions executed on the processor 302; it may be the result of the previous instruction executed at the processor 302 for access by the subsequent instructions executed on the processor 302 or for writing to the memory 304 or the storage 306; or other suitable data. The data cache may speed up the read or write operation of the processor 302. The TLB may speed up the virtual address translation of the processor 302. In some non-limiting embodiments, the processor 302 may include one or more internal registers for data, instructions, or addresses. The present disclosure contemplates that the processor 302 includes any suitable number of any suitable internal registers, where appropriate. Where appropriate, the processor 302 may include one or more arithmetic logic units (Arithmetic Logic Unit, ALU); may be a multi-core processor; or may include one or more processors 302. Although the present disclosure describes and illustrates a specific processor, the present disclosure contemplates any suitable processor.

[0107] In some non-limiting embodiments, memory 304 includes main memory for storing instructions for processor 302 to execute or data for processor 302 to operate. By way of example and not limitation, computer system 300 may load instructions from storage 306 or another source (e.g., another computer system 300) to memory 304. Processor 302 may then load instructions from memory 304 to an internal register or internal cache. To execute the instructions, processor 302 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor 302 may write one or more results (which may be intermediate results or final results) to the internal register or internal cache. Processor 302 may then write one or more of these results to memory 304. In some non-limiting embodiments, processor 302 only executes instructions in one or more internal registers or internal caches or memory 304 (rather than storage 306 or elsewhere), and only operates on data in one or more internal registers or internal caches or memory 304 (rather than storage 306 or elsewhere). One or more memory buses (each of which may include an address bus and a data bus) may couple the processor 302 to the memory 304. The bus 412 may include one or more memory buses, as described below. In certain non-limiting embodiments, one or more memory management units (MMUs) exist between the processor 302 and the memory 304 and facilitate access to the memory 304 requested by the processor 302. In certain other non-limiting embodiments, the memory 304 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port or multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, the memory 304 may include one or more memories 304. Although the present disclosure describes and illustrates specific memories, the present disclosure contemplates any suitable memory.

[0108] In some non-limiting embodiments, storage 306 includes a large capacity memory for data or instructions. As an example and not limitation, storage 306 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more thereof. Where appropriate, storage 306 may include a removable or non-removable (or fixed) medium. Where appropriate, storage 306 may be inside or outside the computer system 300. In some non-limiting embodiments, storage 306 is a non-volatile solid-state memory. In some non-limiting embodiments, storage 306 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically variable ROM (EAROM) or a flash memory, or a combination of two or more thereof. The present disclosure contemplates a large capacity storage 306 in any suitable physical form. Where appropriate, storage 306 may include one or more storage control units that facilitate communication between processor 302 and storage 306. Where appropriate, storage 306 may include one or more storages 306. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0109] In certain non-limiting embodiments, the I / O interface 308 includes hardware, software, or both, which provides one or more interfaces for the communication between the computer system 300 and one or more I / O devices. Where appropriate, the computer system 300 may include one or more of these I / O devices. One or more of these I / O devices may enable communication between a person and the computer system 300. As an example and not limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a static camera, a stylus, a tablet computer, a touch screen, a tracking ball, a camera, other suitable I / O devices, or a combination of two or more thereof. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 308 for it. Where appropriate, one or more device or software drivers may be included so that the processor 302 can drive one or more of these I / O devices. Where appropriate, the I / O interface 308 may include one or more I / O interfaces 308. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0110] In some non-limiting embodiments, the communication interface 410 includes hardware, software, or both, which provides one or more interfaces for communication (e.g., packet-based communication) between the computer system 300 and one or more other computer systems 300 or one or more networks. As an example and not limitation, the communication interface 410 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other wired network, or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network (e.g., a WI-FI network). The present disclosure contemplates any suitable network and any suitable communication interface 410 for it. As an example and not limitation, the computer system 300 may communicate with one or more parts of a self-organizing network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more thereof. One or more parts of one or more of these networks may be wired or wireless. As an example, computer system 300 may communicate with a wireless PAN (WPAN) (e.g., a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more thereof. Computer system 300 may include any suitable communication interface 410 for any of these networks, where appropriate. Communication interface 410 may include one or more communication interfaces 410, where appropriate. Although the present disclosure describes and illustrates particular communication interfaces, the present disclosure contemplates any suitable communication interface.

[0111] In certain non-limiting embodiments, bus 412 includes hardware, software, or both that couples components of computer system 300 to each other. By way of example and not limitation, bus 412 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front-Side Bus (FSB), a Hypertransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBAND interconnect, a Low-Pin-Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCI-Express, PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable busses, or a combination of two or more thereof. Where appropriate, bus 412 may include one or more buses 412. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0112] Herein, where appropriate, the one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more thereof. Where appropriate, the computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0113] As used herein, unless expressly stated otherwise or the context indicates otherwise, "or" is inclusive rather than exclusive. Thus, as used herein, "A or B" means "A, B, or both," unless expressly stated otherwise or the context indicates otherwise. Furthermore, "and" is both conjunctive and separate, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, conjunctive or separate," unless expressly stated otherwise or the context indicates otherwise.

[0114] The scope of the present disclosure covers all changes, substitutions, variations, changes and modifications to the example embodiments described or illustrated herein that will be understood by a person of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or illustrated herein. In addition, although the present disclosure describes and illustrates various embodiments herein as including specific components, elements, features, functions, operations or steps, any of these embodiments may include any combination or arrangement of any components, elements, features, functions, operations or steps that will be understood by a person of ordinary skill in the art and described or illustrated anywhere herein. In addition, the device, or system, or component of the device or system mentioned in the attached claims is adapted, arranged, capable, configured, enabled, operable or operated to perform a specific function, including the device, system, component, whether or not it or the specific function is activated, turned on or unlocked, as long as the device, system or component is so adapted, arranged, capable, configured, enabled, operable or operated. In addition, although the present disclosure describes or illustrates some non-limiting embodiments as providing specific advantages, certain non-limiting embodiments may not provide these advantages, provide some advantages or all of these advantages.

[0115] In addition, the embodiments of the methods presented and described as flow charts in the present disclosure are provided by way of example in order to provide a more complete understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of the various operations is changed and in which sub-operations described as part of a larger operation are performed independently.

[0116] Although various embodiments have been described for the purposes of this disclosure, such embodiments should not be considered to limit the teachings of this disclosure to these embodiments. Various changes and modifications may be made to the above elements and operations to obtain results that remain within the scope of the systems and processes described in this disclosure.

[0117] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited thereto. Certain non-limiting embodiments may include all, some or none of the components, elements, features, functions, operations or steps of the embodiments disclosed above. The attached claims specifically disclose embodiments for methods, storage media, systems and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or references in the attached claims are only for formal reasons. However, any subject matter that is intentionally referenced to any previous claim (especially multiple dependencies) may also be claimed, and therefore, regardless of which dependencies are selected in the attached claims, any combination of claims and their features may be disclosed and claimed. The subject matter that may be claimed includes not only the combination of features listed in the attached claims, but also any other combination of features in the claims, wherein the individual features mentioned in the claims may be combined with any other features or combinations of other features in the claims. In addition, any embodiment and feature described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or with any feature of the attached claims.

[0118] ***

[0119] All patents, patent applications, publications, product descriptions, and protocols cited in this specification are incorporated herein by reference in their entirety. In the event of a conflict in terminology, the present disclosure controls.

[0120] Although it is clear that the subject matter described herein is intended to achieve the above-mentioned benefits and advantages, the subject matter disclosed herein is not limited in scope to the specific embodiments described herein. It should be understood that the disclosed subject matter can be modified, changed and altered without departing from its spirit. Using only routine experimentation, those skilled in the art will recognize or be able to ascertain many equivalents to the specific embodiments described herein. Such equivalents are intended to be covered by the following claims.

[0121] Various references are cited throughout this document, the entire contents of which are incorporated herein by reference.

Claims

1. A method comprising, by one or more computing systems: accessing a plurality of radiographic images of an animal, wherein one or more first radiographic images of the plurality of radiographic images respectively depict the animal from one or more viewing angles, and wherein one or more second radiographic images of the plurality of radiographic images respectively depict one or more body parts of the animal; determining one or more disease classifications associated with the animal based on analyzing the plurality of radiographic images using a machine learning model; generating a diagnostic report associated with the animal based on the machine learning model, wherein the diagnostic report includes the one or more disease classifications and a natural language text radiology report; and An instruction for presenting the diagnostic report is sent to a user device.

2. The method according to claim 1, wherein: Each of the plurality of radiographic images is formatted as a Digital Imaging and Communications in Medicine (DICOM) image.

3. The method according to claim 1, wherein: The machine learning model is based on at least one first neural network and at least one second neural network, and the at least one first neural network and the at least one second neural network are coupled to each other.

4. The method according to claim 1, wherein: Generating the diagnostic report includes: Access multiple reference reports; encoding the plurality of reference reports into a feature space; encoding the plurality of radiographic images into the feature space; and The diagnosis report is determined based on a similarity search in the feature space.

5. The method according to claim 1, wherein: A disease classification of the one or more disease classifications is indicative of abnormal tissue.

6. The method according to claim 5, further comprising: The abnormal tissue is identified as at least one of a cardiovascular, a pulmonary structure, a mediastinal structure, a pleural cavity, or an extra-thoracic cavity.

7. The method according to claim 1, further comprising: accessing a plurality of training radiographic images, wherein the plurality of training radiographic images are respectively associated with a plurality of training radiology reports; and The machine learning model is trained based on the accessed training radiological images and their respective training radiology reports.

8. The method according to claim 7, further comprising: Each of the plurality of training radiographic images is preprocessed, wherein the preprocessing comprises one or more of padding, random enhancement, random flipping, Gaussian blurring or normalization.

9. The method according to claim 7, further comprising: Long document encoding is applied to each of the plurality of training radiology reports.

10. The method according to claim 7, further comprising: Each of the plurality of training radiology reports is preprocessed, wherein the preprocessing includes one or more of tokenizing, padding, adding classification tags, or applying an attention mask.

11. The method according to claim 1, wherein: The machine learning model includes an image encoder, a multi-image encoder, a text decoder, and a multimodal decoder.

12. The method according to claim 11, further comprising: generating, by the image encoder, a feature map based on the plurality of radiographic images; generating, by the multi-image encoder, one or more multi-image keys and values ​​based on the feature map; and The natural language textual radiology report is generated by the multimodal decoder based on the one or more multi-image keys and values ​​and sentence start markers.

13. The method according to claim 1, wherein: The diagnostic report also includes one or more radiological image images among the plurality of radiological image images.

14. One or more computer-readable non-transitory storage media containing software that, when executed, is operable to: accessing a plurality of radiographic images of an animal, wherein one or more first radiographic images of the plurality of radiographic images respectively depict the animal from one or more perspectives, and one or more second radiographic images of the plurality of radiographic images respectively depict one or more body parts of the animal; determining one or more disease classifications associated with the animal based on analyzing the plurality of radiographic images using a machine learning model; generating a diagnostic report associated with the animal based on the machine learning model, wherein the diagnostic report includes the one or more disease classifications and a natural language text radiology report; and An instruction for presenting the diagnostic report is sent to a user device.

15. The medium according to claim 14, wherein Each of the plurality of radiographic images is formatted as a Digital Imaging and Communications in Medicine (DICOM) image.

16. The medium according to claim 14, wherein The machine learning model is based on at least one first neural network and at least one second neural network, which are coupled to each other.

17. The medium according to claim 14, wherein: Generating the diagnostic report includes: Access multiple reference reports; encoding the plurality of reference reports into a feature space; encoding the plurality of radiographic images into the feature space; and The diagnosis report is determined based on a similarity search in the feature space.

18. The medium according to claim 14, wherein A disease classification of the one or more disease classifications is indicative of abnormal tissue.

19. The medium according to claim 18, wherein The software, when executed, is further operable to: The abnormal tissue is identified as at least one of a cardiovascular, a pulmonary structure, a mediastinal structure, a pleural cavity, or an extra-thoracic cavity.

20. The medium of claim 14, wherein: The software, when executed, is further operable to: accessing a plurality of training radiographic images, wherein the plurality of training radiographic images are respectively associated with a plurality of training radiology reports; and The machine learning model is trained based on the accessed training radiological images and their respective training radiology reports.

21. The medium according to claim 20, wherein The software, when executed, is further operable to: Each of the plurality of training radiographic images is preprocessed, wherein the preprocessing comprises one or more of padding, random enhancement, random flipping, Gaussian blurring or normalization.

22. The medium according to claim 20, wherein The software, when executed, is further operable to: Long document encoding is applied to each of the plurality of training radiology reports.

23. The medium of claim 20, wherein: The software, when executed, is further operable to: Each of the plurality of training radiology reports is preprocessed, wherein the preprocessing includes one or more of tokenizing, padding, adding classification tags, or applying an attention mask.

24. The medium of claim 14, wherein: The machine learning model includes an image encoder, a multi-image encoder, a text decoder, and a multimodal decoder.

25. The medium of claim 24, wherein: The software, when executed, is further operable to: generating, by the image encoder, a feature map based on the plurality of radiographic images; generating, by the multi-image encoder, one or more multi-image keys and values ​​based on the feature map; and The natural language textual radiology report is generated by the multimodal decoder based on the one or more multi-image keys and values ​​and sentence start markers.

26. The medium of claim 14, wherein: The diagnostic report also includes one or more radiological image images among the plurality of radiological image images.

27. A system comprising: one or more processors; and a non-transitory memory coupled to the processor, the non-transitory memory comprising instructions executable by the processor, the processor, when executing the instructions, being operable to: accessing a plurality of radiographic images of an animal, wherein one or more first radiographic images of the plurality of radiographic images respectively depict the animal from one or more perspectives, and one or more second radiographic images of the plurality of radiographic images respectively depict one or more body parts of the animal; determining one or more disease classifications associated with the animal based on analyzing the plurality of radiographic images using a machine learning model; generating a diagnostic report associated with the animal based on the machine learning model, wherein the diagnostic report includes the one or more disease classifications and a natural language text radiology report; and An instruction for presenting the diagnostic report is sent to a user device.

28. The system of claim 27, wherein: Each of the plurality of radiographic images is formatted as a Digital Imaging and Communications in Medicine (DICOM) image.

29. The system of claim 27, wherein: The machine learning model is based on at least one first neural network and at least one second neural network, which are coupled to each other.

30. The system of claim 27, wherein: Generating the diagnostic report includes: Access multiple reference reports; encoding the plurality of reference reports into a feature space; encoding the plurality of radiographic images into the feature space; and The diagnosis report is determined based on a similarity search in the feature space.

31. The system of claim 27, wherein: A disease classification of the one or more disease classifications is indicative of abnormal tissue.

32. The system of claim 31, wherein: The processor, when executing the instructions, is further operable to: The abnormal tissue is identified as at least one of a cardiovascular, a pulmonary structure, a mediastinal structure, a pleural cavity, or an extra-thoracic cavity.

33. The system of claim 27, wherein: The processor, when executing the instructions, is further operable to: accessing a plurality of training radiographic images, wherein the plurality of training radiographic images are respectively associated with a plurality of training radiology reports; and The machine learning model is trained based on the accessed training radiological images and their respective training radiology reports.

34. The system of claim 33, wherein: The processor, when executing the instructions, is further operable to: Each of the plurality of training radiographic images is preprocessed, wherein the preprocessing comprises one or more of padding, random enhancement, random flipping, Gaussian blurring or normalization.

35. The system of claim 33, wherein: The processor, when executing the instructions, is further operable to: Long document encoding is applied to each of the plurality of training radiology reports.

36. The system of claim 33, wherein: The processor, when executing the instructions, is further operable to: Each of the plurality of training radiology reports is preprocessed, wherein the preprocessing includes one or more of tokenizing, padding, adding classification tags, or applying an attention mask.

37. The system of claim 27, wherein: The machine learning model includes an image encoder, a multi-image encoder, a text decoder, and a multimodal decoder.

38. The system of claim 37, wherein: The processor, when executing the instructions, is further operable to: generating, by the image encoder, a feature map based on the plurality of radiographic images; generating, by the multi-image encoder, one or more multi-image keys and values ​​based on the feature map; and The natural language textual radiology report is generated by the multimodal decoder based on the one or more multi-image keys and values ​​and sentence start markers.

39. The system of claim 27, wherein: The diagnostic report also includes one or more radiological image images among the plurality of radiological image images.

Images