Method and system with additive AI model

Abstract

A system and method for obtaining additive AI results from digital files is described, the method including processing the digital files with at least one first artificial intelligence (AI) classifier and at least one second AI classifier, thereby obtaining a first evaluation result and at least one second evaluation result, respectively; directing the first evaluation result and the at least one second evaluation result to at least one synthesis processor; and comparing the first evaluation result and the at least one second evaluation result with at least one dataset cluster, thereby obtaining additive AI results.

Description

[Technical Field]

[0001] This application claims benefit of and priority to U.S. Provisional Patent Application No. 62 / 954,046, filed December 27, 2019, and U.S. Provisional Patent Application No. 62 / 980,669, filed February 24, 2020, both entitled "Efficient Artificial Intelligence Analysis of Images," by inventors Seth Wallack, Ariel Ayaviri Omonte, and Ruben Venegas, and to U.S. Provisional Patent Application No. 63 / 083,422, filed September 25, 2020, entitled "Efficient artificial intelligence analysis of images with combined predictive modeling," by inventors Seth Wallack, Ariel Ayaviri Omonte, Ruben Venegas, Yuan-Ching Spencer Teng, and Pratheev Sabaratnam Sreetharan. This application claims the benefit of U.S. Provisional Patent Application No. 63 / 395,525, filed August 5, 2022, by inventors Seth Wallack and Eric Goldman, entitled "Additive AI classifiers," which is a continuation-in-part of U.S. Utility Patent Application No. 17 / 134,990, filed December 28, 2020, which is a continuation-in-part of International Application No. PCT / US20 / 66580, filed December 22, 2020, by inventors Seth Wallack, Yuan-Ching Spencer Teng, and Pratheev Sabaratnam Sreetharan, entitled "Efficient artificial intelligence analysis of images with combined predictive modeling," which is incorporated herein by reference in its entirety. [Background technology]

[0002] background Artificial intelligence (AI) processors, e.g., trained neural networks, are useful for processing radiological images of animals to determine the probability that the imaged animal has a particular condition. Typically, separate AI processors are used to evaluate each body part (e.g., chest, abdomen, shoulders, forelimbs, hindlimbs, etc.) and / or a particular orientation of each such body part (e.g., ventrodorsal (VD) view, lateral view, etc.). A particular AI processor, for each body part and / or orientation, determines the probability that a particular condition exists for the body part in question. Each such AI processor contains multiple trained models that evaluate each condition or organ within the imaged region. For example, for a lateral view of an animal's chest, the AI ​​processor employs different models to determine the probability that the animal has a particular lung-related condition (e.g., perihilar infiltrates, pneumonia, bronchitis, pulmonary nodules, etc.).

[0003] The amount of processing performed by each such AI processor, and the amount of time required to complete such processing, is enormous. This task requires either (1) manual identification and cropping of each image to define a specific body part and orientation before the image is evaluated by a particular AI processor, or (2) feeding the images to each AI processor for evaluation. Unlike human radiology, where radiological examinations are limited to specific body parts, veterinary radiology routinely includes multiple unlabeled images with multiple body parts of unknown orientation within a single examination.

[0004] In a conventional workflow for processing animal radiology images, the system assumes that the image contains a body part identified by the user. The user-identified image is then sent to a specific AI processor that evaluates the probability of the presence of a pathology for that particular body part, for example, using a machine learning model. However, requiring the user to identify the body part creates friction in the conventional workflow and leads to errors if the identified body part is inaccurate or if multiple regions are included in the image. In addition, the conventional workflow becomes inefficient (or breaks down) when images without user identification of the body part are sent to the system. When this occurs, the conventional workflow is inefficient because unidentified images are sent to multiple AI processors that are not specialized for the imaged body part. Furthermore, the conventional workflow is prone to erroneous results because incorrect body part identification results in images being sent to AI processors configured to evaluate different body parts.

[0005] A conventional workflow for analyzing diagnostic features of radiographs using AI and generating a report based on the AI ​​model diagnosis results in an exponential number of possible output reports. The AI ​​model diagnosis result provides an identification of either normal or abnormal for a particular condition. In some AI models, an identification of the severity of a particular condition, e.g., normal, minimal, mild, moderate, or severe, is also provided. The set of AI model diagnosis results specifies which report should be selected from pre-created report templates. The process of creating and selecting a single report template from a set of AI model diagnosis results scales exponentially with the number of AI models. Six different AI model normal / abnormal diagnosis results require 64 different report templates (2 to the power of 6). Ten models require 1,024 templates, and 16 models require 65,536 templates. An AI model that detects the severity scale even more poorly, e.g., 16 severity detection models with five possible severity levels each, would require over 150 billion templates. Therefore, manually generated reports for each combination of AI model diagnostic results do not scale well to multiple AI models being interpreted together.

[0006] Therefore, there is a need for a novel system that has several fully automated stages of image pre-processing and image analysis, including identifying whether a received image contains a particular body part in a particular orientation (e.g., a lateral view), appropriately cropping the image, creating one or more sub-images from an original image containing two or more body parts or regions of interest, labeling the original image and any created sub-images, and evaluating the cropped image and sub-images against a targeted AI model. Additionally, there is a need for a novel system that analyzes and provides a diagnostic radiologist report based on multiple examination results, including, but not limited to, AI model results.

[0007] In machine learning, AI models or classifiers are continuously trained to improve model performance. Best practices for AI model deployment involve continuous training, including retraining currently deployed AI models. Retraining is based on one or more of these common parameters: AI performance-based, triggered by data changes, or training on demand. The current industry standard for retraining AI models aims to replace the currently trained model with a new, "improved" AI model. Current AI technology is based on a single model result in a production environment, thus evolving the concept of retraining and replacement.

[0008] The problem with the current approach is that replacing the current AI model with a newly trained AI model operates under the assumption that improving model performance reduces unnecessary "noise" or false positive data. Data detected as unnecessary "noise" in the current AI model is removed and replaced with the retrained AI model. However, the assumption that "noise" is not valuable information in overall system performance is incorrect. The "noise" detected by the current model classifier is useful for distinguishing data points with similar but not identical characteristics. Therefore, by replacing the current AI model, data is lost.

[0009] Therefore, there is a need for a new approach to the continuous training of AI models that incorporates both new and old AI models. Summary of the Invention [Means for solving the problem]

[0010] overview One aspect of the invention described herein provides a method for obtaining additive AI results from a digital file, the method including: processing the digital file with a first artificial intelligence (AI) classifier and at least one second AI classifier, thereby obtaining a first evaluation result and at least one second evaluation result, respectively; directing the first evaluation result and the at least one second evaluation result to at least one synthesis processor; and comparing the first evaluation result and the at least one second evaluation result with at least one dataset cluster, thereby obtaining an additive AI result. The terms AI model and AI classifier are used interchangeably and are defined as a type of machine learning algorithm used to assign class labels to data inputs.

[0011] An embodiment of the method further includes measuring a distance from the additive AI result to an example result from the dataset cluster to obtain an additive AI cluster identification. In an embodiment of the method, the dataset cluster further includes a matched pre-written template. An embodiment of the method further includes assembling the additive AI cluster identification and the matched pre-written template to obtain a report. An embodiment of the method further includes displaying the report to a user.

[0012] In one embodiment of the method, the second AI classifier is a derivative of the first AI classifier. In one embodiment of the method, the second AI classifier is trained using at least a portion of the data used to train the first AI classifier. For example, the second classifier is trained using at least 99%, 95%, 90%, 85%, 80%, 75%, 70%, 66%, 60%, 55%, 50%, 45%, 40%, 35%, 30%, 25%, 20%, 15%, 10%, or 5% of the data used to train the first AI classifier. In one embodiment of the method, the second AI classifier is related to the first AI classifier. In some embodiments, the first AI classifier is a generic classifier. In some embodiments, the second AI classifier is a specific classifier. In an alternative embodiment, the first AI classifier is a specific classifier and the second AI classifier is a generic classifier.

[0013] An embodiment of the method further includes repeating the training and comparing steps for a series of daisy-chained AI classifiers. An embodiment of the method further includes comparing the first evaluation results with the second evaluation results to train the first and second AI classifiers or to compare the AI ​​results and test expected performance.

[0014] An embodiment of the method further includes adding the first assessment result and the second assessment result to a results database. An embodiment of the method further includes obtaining the digital file before processing. An embodiment of the method further includes converting the analog file to a digital file before processing. An embodiment of the method further includes classifying the digital file by performing at least one of labeling, cropping, editing, and orienting the digital file before processing.

[0015] An embodiment of the method further includes adjusting the AI ​​model result in a heuristic manner by applying a mathematical formula to the AI ​​model result, the mathematical formula including at least one of addition, subtraction, multiplication, division, or other standard mathematical formula.

[0016] One aspect of the invention described herein provides a system programmed to obtain an additive AI result by any of the methods described herein, the system including at least one first AI processor, at least one derivative AI processor derived from the first AI processor, and an output device.

[0017] An embodiment of the system further includes at least one database library. An embodiment of the system further includes a user interface. In some embodiments, the AI ​​results from one derived classifier for only one first AI classifier. In some embodiments, more than one derived classifier for only one first AI classifier is included in the results. In some embodiments, one derived classifier for more than one first AI classifier is included in the results. In some embodiments, more than one derived classifier for more than one first AI classifier is included in the results. In some embodiments, the results include one derived classifier for only some of the first AI classifiers and more than one derived classifier for some of the first AI classifiers. [Brief explanation of the drawings]

[0018] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] 1 is a schematic diagram of a conventional workflow for processing a radiological image 102 of an animal. As commonly expressed in the veterinary field, the image 102 does not show any parts of the animal. The image 102 is processed by each of multiple AI processors 104a-104l to determine whether that body part is present on the image and the probability that the animal body depicted in the image 102 has a particular condition. Each of the AI ​​processors 104a-104l evaluates the image 102 by comparing it to one or more machine learning models, each trained to determine the probability that the animal has a particular condition. [Figure 2]1 is a schematic diagram of one embodiment of a system or method described herein. A radiology image preprocessor 106 is deployed to preprocess an image 102 to generate one or more sub-images 108, each corresponding to a particular view of a particular body part. Three sub-images 108a-c are generated, with one sub-image 108a identified and cropped as a lateral view of the animal's thorax, a second sub-image 108b identified and cropped as a lateral view of the animal's abdomen, and a third sub-image 108c identified and cropped as a lateral view of the animal's pelvis. As shown, sub-image 108a is processed only by the thorax-lateral AI processor 104a, sub-image 108b is processed only by the abdomen-lateral AI processor 104c, and sub-image 108c is processed only by the pelvis-lateral AI processor 104k. In some embodiments, the sub-images 108 are tagged to identify the body part and / or view that the sub-image 108 represents. [Figure 3] 1 is a diagram illustrating a set of computer operations performed by one embodiment of the system or method of the present invention herein for the novel workflow described herein. Image 302 is processed using the radiology image preprocessor 106 and then the subset of AI processor 104 corresponding to the identified body parts / views. Cropped images 304a, 304b of each body part / view identified by the system are shown. The total time spent by the radiology image preprocessor to identify image 302 as representing both a "lateral chest" image and a "lateral abdomen" image was 24 seconds, as reflected by the timestamp of the log entry corresponding to bracket 306. [Figure 4] A set of conventional single-condition basic organ findings 401-407 for lungs on radiographs, followed by combinations of at least two single-condition basic organ findings. The permutations and combinations of the seven single-condition basic organ findings result in an exponential number of report templates. [Figure 5A]Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5B] Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5C] Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5D] Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5E] Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5F]Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5G] Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5H] Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5I] Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5J] Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5K]Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5L] Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5M] Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 5N] Pulmonary basic organ findings classified based on severity as normal, minimal, mild, moderate, and severe and displayed as separate AI model result templates. Boxes 501-557 represent single line items in a particular AI report template. A single line item is selected based on each AI model result template matching the finding listed under the heading "Code." [Figure 6] A collection of individual binary models (or a library of AI models) that are deployed to analyze radiological images and obtain a probability result that the radiological image is negative or positive for a condition or classification. [Figure 7] A lateral chest radiograph of a dog has been preprocessed, cropped, labeled, and identified. The radiograph is then analyzed by a library of binary AI models, shown in Figure 6. [Figure 8] 8 is a screenshot of the results of a single binary AI model obtained by analyzing a series of lateral radiographic images similar to the images of FIG. 7 through a particular binary AI model, e.g., the bronchitis AI model. [Figure 9A] 9A is a screenshot showing each AI model result for a radiology image. A visual collection of each individual AI model result for each image and the average AI model result for all images evaluated for that particular case. The average evaluation result for each model is created by assembling the individual image evaluation results and is displayed at the top of the screen in FIG. 9A. The timestamp 901 in FIG. 9A indicates that the AI ​​analysis was completed in less than 3 minutes. [Figure 9B] 9A and 9B are screenshots showing the results of each AI model for a radiology image. A visual collection of each individual AI model result for each image and the average AI model result for all images evaluated for that particular case is displayed in FIG. 9B. FIG. 9B shows the results for each individual AI model, including perihilar infiltrates, pneumonia, bronchitis, interstitial, diseased lung, hypoplastic trachea, cardiomegaly, pulmonary nodule, and pleural effusion. For each AI model, a label identifies the image as "normal" 902 or "abnormal" 903. Additionally, the probability 904 that the image is "normal" or "abnormal" for each AI model condition is provided. [Figure 9C] 9A-9C are screenshots showing the AI ​​model results for each radiology image. A visual collection of each individual AI model result for each image and the average AI model result for all images evaluated for that particular case. Each individual image and the AI ​​model results for that image are displayed in FIG. 9C. FIG. 9C shows four images obtained by classifying and cropping a single radiology image. Time stamps 905-908 indicate that the AI ​​analysis was completed in less than two minutes. [Figure 9D]9D is a screenshot showing each AI model result for a radiology image. A visual collection of each individual AI model result for each image and the average AI model result for all images evaluated for that particular case. Each individual image and the AI ​​model result for that image are displayed in FIG. 9D. FIG. 9D shows the results of each AI model for the radiology image, including the radiology image label, probability, and view (909), e.g., lateral, dorsal, anterior-posterior, posteroanterior, ventrodorsal, dorsoventral, etc. [Figure 9E] 9E is a screenshot showing each AI model result for a radiology image. A visual collection of each individual AI model result for each image and the average AI model result for all images evaluated for that particular case. Each individual image and the AI ​​model result for that image are displayed in FIG. 9E. FIG. 9E shows the results of each AI model for the radiology image, including the radiology image label, probability, and view (909), e.g., lateral, dorsal, anterior-posterior, posteroanterior, ventrodorsal, dorsoventral, etc. [Figure 10] This is a screenshot of the AI ​​case results displayed in JavaScript Object Notation (JSON) format. The JSON format makes it easy to copy the average evaluation results of all models, which can then be transferred to an AI evaluation tester for testing to evaluate the average evaluation results by comparing them with the cluster results. [Figure 11]11 is a screenshot of a graphical user interface that allows a user to create K-means clusters. The user assigns a name 1101 for the new cluster under "Code." The user selects various parameters to create the cluster. The user selects cases by selecting a case start date 1102 and a case end date 1103. The user selects cases by selecting a case start ID 1104 and a case end ID 1105. The user selects the maximum number of cases 1106 to be included in the cluster. The user selects a species 1107, such as canine, feline, canine or feline, human, etc., for the cases to be included in the cluster. The user selects specific diagnostic modalities 1108, such as x-ray, CT, MRI, blood analysis, urinalysis, etc., to be included in the creation of the cluster. The user specifies that the assessment results be separated into a specific number of clusters. The number of clusters ranges from a minimum of one cluster to a maximum number of clusters limited only by the total number of cases input into the cluster. [Figure 12A] 12 is a screenshot of AI cluster results listed as a table of values. The leftmost column 1201 is the case ID, the next nine columns are the average evaluation results 1202 of each binary model for the particular case ID, the next column is the cluster label or cluster location 1203 that contains the particular case based on the set of evaluation results, the next four columns are the cluster coordinates and centroid coordinates, and the last number is the case ID 1204 of the centroid or center of that particular cluster. A radiologist report is obtained for the best-matching case ID. This radiologist report is then used to generate a report for the new AI case. This process allows for infinite scalability in terms of the number of AI models incorporated compared to traditional semi-manual report creation processes. [Figure 12B]12 is a screenshot of AI cluster results listed as a table of values. The leftmost column 1201 is the case ID, the next nine columns are the average evaluation results 1202 of each binary model for the particular case ID, the next column is the cluster label or cluster location 1203 that contains the particular case based on the set of evaluation results, the next four columns are the cluster coordinates and centroid coordinates, and the last number is the case ID 1204 of the centroid or center of that particular cluster. A radiologist report is obtained for the best-matching case ID. This radiologist report is then used to generate a report for the new AI case. This process allows for infinite scalability in terms of the number of AI models incorporated compared to traditional semi-manual report creation processes. [Figure 13] 11 is an example of a clustering graph. The clustering graph is created by dividing the average rating results into multiple distinct clusters according to user-defined parameters 1102-1108. This exemplary clustering graph is divided into 180 distinct clusters, each represented by a neighborhood of single-colored dots plotted on the graph. [Figure 14] 14 is a screenshot of a user interface showing AI cluster models generated based on user-defined parameters 1102-1108. The first column from the left indicates the cluster ID 1401, the second column indicates the assigned name 1402 of the cluster model, the third column indicates the number of different clusters 1403 into which the AI ​​data was divided, and the fourth column indicates the body part 1404 that was evaluated based on the cluster data results. [Figure 15] 15 is a screenshot of a user interface showing a screening evaluation configuration. The user interface allows for the assignment of a specific "cluster model" 1502 to a specific "screening evaluation configuration" name 1501. The status 1503 of the screening evaluation configuration provides additional data about the configuration, such as whether the configuration is in live, test, or draft mode. Live mode is for production, and test mode is for development. [Figure 16A]16A shows a screenshot of a user interface showing details of a particular cluster model. FIG. 16A shows a user interface displaying data for a cluster model 1601 of the breast 97. The AI ​​assessment classifier types 1602 included in the cluster are listed. The species or species collection 1603 specific to the cluster model is displayed. The maximum number of cases with assessment results used to generate the cluster 1604 is displayed. The user interface shows the start and end dates 1605 for the cases used to create the cluster. A link 1606 to the comma-separated values ​​(CSV) file of FIGS. 12A and 12B showing the cluster in a numeric table format is displayed. A portion 1608 of the subclusters created from parameters 1602-1605 is listed. The total number 1609 of subclusters created for this cluster group is displayed. For each subcluster, the centroid case ID 1610 is displayed. A link 1607 to the log for building the cluster is displayed. [Figure 16B] 16A is a screenshot of a user interface showing details of a particular cluster model. FIG. 16B is a screenshot of a log created for the cluster model 1601 of the thorax 97. [Figure 16C] 16A and 16B are screenshots of the user interface showing details of a particular cluster model. Figure 16C is a screenshot of a portion of the AI ​​assessment model including vertebral heart score, perihilar infiltrates, pneumonia, bronchitis, interstitial, and diseased lung. [Figure 17A] 17A is a screenshot of a user interface for an AI evaluation tester. Figure 17A shows a user interface (AI evaluation tester) in which the average evaluation result values ​​of all models in the JSON format of Figure 10 are imported 1701 to analyze the nearest match case / sample result matches within clusters using K-means clustering from case clusters created from an AI dataset. [Figure 17B]17A and 17B are screenshots of the user interface for the AI ​​evaluation tester, and show the average evaluation result values ​​for all models in the JSON format of FIG. 10 being imported into the AI ​​evaluation tester. [Figure 17C] 17A and 17B are screenshots of the user interface for the AI ​​evaluation tester. FIG. 17C shows the imported evaluation results for a specific case. [Figure 17D] 17A and 17B are screenshots of the user interface for the AI ​​assessment tester. FIG. 17D shows the assessment results imported for a specific case. FIG. 17D shows the screening assessment type 1702 selected by the user and the cluster model 1703 associated with the screening assessment type. By clicking test 1704, the assessment results displayed in FIG. 10 are analyzed and assigned to the nearest matching case / exemplar result match within the cluster. The nearest radiologist report, top-ranked radiologist statement, and centroid radiologist report for the exemplar result match cluster are collected and displayed. [Figure 18A] 18A is a screenshot of a user interface showing the results displayed after clicking a test 1704 on the AI ​​assessment tester. Diagnosis and conclusion findings 1801 from the radiologist reports that most closely match the assessment results based on the previously created cluster results are displayed. Assessment findings 1802 are selected from the radiologist reports within the clusters of assessment results and filtered based on the occurrence of specific sentences within the findings section of the particular cluster. [Figure 18B] 18A and 18B are screenshots of a user interface showing the results displayed after clicking test 1704 on the AI ​​assessment tester. Recommendations 1803 from radiologist reports within a cluster are selected based on the occurrence of each sentence or similar sentences in the recommendations section of this cluster. Interface 1804 shows the radiologist reports for the cluster, and interface 1805 shows the radiologist reports for the centroid of the cluster. [Figure 18C] 18A and 18B are screenshots of a user interface listing the ranking of statements in a radiology report based on a particular cluster result. The statements include a conclusion statement 1806, a finding statement 1807, and a recommendation statement 1808. [Figure 18D] 18D is a screenshot of a user interface that allows a user to edit a radiology report by editing the findings section 1809, conclusions section 1810, or recommendations section 1811 by adding or deleting specific statements. [Figure 19] The radiologist report for the nearest match dataset case is used to generate a radiology report for the new case. The AI ​​assessment tester displays the radiologist report that is closest to the current AI assessment result based on the similarity of the assessment results between the AI ​​assessment result for the new image and the AI ​​assessment results in the cluster and the radiologist report from the centroid of the selected cluster. [Figure 20A] Figure 20A is a newly received radiograph being analyzed. Cluster matching is based on AI assessment results, not image match results. [Figure 20B] Figure 20B is a radiograph selected by the results of AI evaluation as a nearest match based on the cluster model. The cluster match is based on the AI ​​evaluation results, not the image match results. [Figure 21A] 21A is a schematic diagram of components in an AI radiograph processing unit. Figure 21A is a schematic diagram showing a radiographer 2101 sending radiographic images to a desktop application 2102 which routes the images to a web application 2103. A computer vision application 2104 and web application routes the images to an imaging web application which routes the images to an AI evaluation 2105. [Figure 21B]FIG. 21B is a schematic diagram of the components in the AI ​​radiograph processing unit. FIG. 21B is a schematic diagram of the components in the Image Match AI process. Images uploaded to the veterinary clinic's Local Interface to Online Network (LION) 2106 are directed to the VetConsole 2107, which auto-rotates and auto-crops the image to obtain sub-images. The sub-images are directed to three places. The first place is the VetAI console 2108, which classifies the image. The second place is the Image Match console 2109, which adds the sub-images along with a report to the Image Match database. The third place is the Image Database 2110, which stores the new image and corresponding case ID number. The Image Match console 2109 directs the image to the Refinement Image Match console 2111 or the VetImage Editor console 2112 for further processing. [Figure 22A] 22A is a schematic diagram of a server architecture for image matching, and FIG. 22A is a schematic diagram of a server architecture currently used in AI radiograph analysis. [Figure 22B] Figure 22B is a schematic diagram of a server architecture for image matching currently used in AI radiograph analysis. [Figure 22C] 22C is a schematic diagram of a server architecture for image matching. Figure 22C is a schematic diagram of a server architecture for AI radiograph analysis, which includes pre-processing radiographic images, analyzing the images using an AI diagnostic processor, and generating reports based on the clustering results. Images from PC 2201 are directed to NGINX load balancing server 2202, which directs the images to V2 cloud platform 2203. The images are then directed to image match server 2204, VetImages server 2205, and database Microsoft SQL server 2207. VetImages server directs the images to VetAI server 2206, database Microsoft SQL server 2207, and data store server 2208. [Figure 22D]22D is a schematic diagram of a server architecture for image matching. Figure 22D is a schematic diagram of a server architecture for AI radiograph analysis, including pre-processing radiographic images, analyzing the images using an AI diagnostic processor, and generating reports based on the clustering results. Images from PC 2201 are directed to NGINX load balancing server 2202, which directs the images to V2 cloud platform 2203. The images are then directed to image match server 2204, VetImages server 2205, and database Microsoft SQL server 2207. VetImages server directs the images to VetAI server 2206, database Microsoft SQL server 2207, and data store server 2208. [Figure 23A]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The sub-image processor accomplishes tasks 2301-2332 listed in Figures 23A-23J. [Figure 23B]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The sub-image processor accomplishes tasks 2301-2332 listed in Figures 23A-23J. [Figure 23C]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The sub-image processor accomplishes tasks 2301-2332 listed in Figures 23A-23J. [Figure 23D]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The sub-image processor accomplishes tasks 2301-2332 listed in Figures 23A-23J. [Figure 23E]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The sub-image processor accomplishes tasks 2301-2332 listed in Figures 23A-23J. [Figure 23F]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The sub-image processor accomplishes tasks 2301-2332 listed in Figures 23A-23J. [Figure 23G]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The sub-image processor accomplishes tasks 2301-2332 listed in Figures 23A-23J. [Figure 23H]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The sub-image processor accomplishes tasks 2301-2332 listed in Figures 23A-23J. [Figure 23I]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The sub-image processor accomplishes tasks 2301-2332 listed in Figures 23A-23J. [Figure 23J]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The sub-image processor accomplishes tasks 2301-2332 listed in Figures 23A-23J. [Figure 23K]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The evaluation processor performs tasks 2333-2346, 2356, and 2357 listed in Figures 23K-O and portions of Figures 23P and 23S. [Figure 23L]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The evaluation processor performs tasks 2333-2346, 2356, and 2357 listed in Figures 23K-O and portions of Figures 23P and 23S. [Figure 23M]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The evaluation processor performs tasks 2333-2346, 2356, and 2357 listed in Figures 23K-O and portions of Figures 23P and 23S. [Figure 23N]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The evaluation processor performs tasks 2333-2346, 2356, and 2357 listed in Figures 23K-O and portions of Figures 23P and 23S. [Figure 23O]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The evaluation processor performs tasks 2333-2346, 2356, and 2357 listed in Figures 23K-O and parts of Figures 23P and 23S. The synthesis processor performs tasks 2347-2355 and 2358-2363 listed in Figures 23O-23T. [Figure 23P]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The evaluation processor performs tasks 2333-2346, 2356, and 2357 listed in Figures 23K-O and parts of Figures 23P and 23S. The synthesis processor performs tasks 2347-2355 and 2358-2363 listed in Figures 23O-23T. [Figure 23Q]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The synthesis processor performs tasks 2347-2355 and 2358-2363 listed in Figures 23O-23T. [Figure 23R]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The synthesis processor performs tasks 2347-2355 and 2358-2363 listed in Figures 23O-23T. [Figure 23S]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The evaluation processor performs tasks 2333-2346, 2356, and 2357 listed in Figures 23K-O and parts of Figures 23P and 23S. The synthesis processor performs tasks 2347-2355 and 2358-2363 listed in Figures 23O-23T. [Figure 23T]This is a schematic diagram of an artificial intelligence automated cropping and evaluation workflow for images acquired of a subject. The workflow is categorized into six columns based on the platform used to accomplish the task: Clinic, V2 end-user application, VetImages web application, VetConsole python script application, VetAI machine learning application, ImageMatch orientation python application, and ImageMatch validation python application. Furthermore, tasks are shaded differently based on the processor that accomplishes the task: sub-image processor, evaluation processor, and synthesis processor. The V2 application is the end-user application where the user interacts with the application and uploads the images to be analyzed. The VetImages application processes the images and generates AI results, AI reports, or evaluation results. VetConsole is a python script application that enhances image quality and processes images in bulk. VetAI is a machine learning application that creates an AI model and evaluates images input into the system. ImageMatch orientation is a python application that searches for correctly oriented images in its database that are similar to the input image. ImageMatch validation is a python application that searches for correctly classified images in its database that are similar to the input image. The synthesis processor performs tasks 2347-2355 and 2358-2363 listed in Figures 23O-23T. [Figure 24] 1 is a diagram of the current model showing the image classifiers being successively replaced, with the oldest model at the top and the most recent model at the bottom. [Figure 25] A new model is presented by the claimed approach in which derived image classifiers are used together rather than replaced, where I denotes the same image classifier and the number or N denotes the derived classifiers, where N represents any positive integer. [Figure 26]A specific embodiment is shown in which the text version can be associated with daisy-chained derived classifiers creating an n:1 relationship. [Figure 27] We show a particular embodiment in which multiple text versions can be associated with daisy-chained derived classifiers creating n:n relationships. [Figure 28] Visual examples of both derived (same letter and subscript number) and non-derived (same letter but different subscript) data are shown together. The database holds all of the training data with relationships. The system allows data input (in this case, I, T, and OI), single or clustered, AI or non-AI data, derived or non-derived data to obtain example results from the database. [Figure 29A] Shown is a chest radiograph of a cat with minimal lung disease. The GLC result is 0.68 and the GLC2 result is 0.69. Together these results are used as a check and balance system to validate the results of each classifier. [Figure 29B] Shown here is a chest radiograph from a cat with moderate lung lesions, specifically a moderate bronchial pattern. Both GLC and its derivative classifier, GLC2, are trained to identify the bronchial pattern in the radiograph. The result for GLC is 0.85, and the result for GLC2 is 0.78. These results are used together as a check and balance system to ensure that the classifier results are true positives. DETAILED DESCRIPTION OF THE INVENTION

[0019] Detailed Description One aspect of the invention described herein provides a method for analyzing diagnostic radiological images or images of a subject, the method including automatically processing the radiological images of the subject using a processor to classify the images into one or more body parts or body regions, orienting and cropping the classified images to obtain at least one oriented, cropped and labeled sub-image for each automatically classified body part, directing the sub-images to at least one artificial intelligence processor, and evaluating the sub-images by the artificial intelligence processor, thereby analyzing the radiological images of the subject.

[0020] An embodiment of the method further includes using an artificial intelligence processor to evaluate the sub-images for a body part and for the presence of a pathology, such as a chest, abdomen, forelimbs, hind limbs, etc. An embodiment of the method further includes using an artificial intelligence processor to diagnose a pathology from the sub-images. An embodiment of the method further includes using an artificial intelligence processor to evaluate the sub-images for positioning of the subject. An embodiment of the method further includes correcting the positioning of the subject to an appropriate positioning.

[0021] In one embodiment of the method, the processor automatically rapid processes the radiographic image to obtain the sub-images. In one embodiment of the method, the processor processes the radiographic image to obtain the sub-images in less than about 1 minute, less than about 30 seconds, less than about 20 seconds, less than about 15 seconds, less than about 10 seconds, or less than about 5 seconds. In one embodiment of the method, the evaluating further includes comparing the sub-images to a plurality of reference radiographic images in at least one of a plurality of libraries. In one embodiment of the method, each of the plurality of libraries includes a respective plurality of reference radiographic images.

[0022] In one embodiment of the method, each of the plurality of libraries includes a respective plurality of reference radiographic images, which may be species-specific or non-specific. An embodiment of the method further includes matching the sub-image with the reference radiographic images to thereby assess orientation and at least one body part. In one embodiment of the method, the reference radiographic images are oriented in a Digital Imaging and Communication in Medicine (DICOM) standard hanging protocol.

[0023] In one embodiment of the method, cropping further includes isolating specific body parts within the sub-images. In one embodiment of the method, cropping further includes classifying the reference radiology images according to veterinary radiology standard body part labels. In one embodiment of the method, orienting further includes adjusting the radiology images to a veterinary radiology standard hanging protocol. In one embodiment of the method, cropping further includes cropping the radiology sub-images to a standard aspect ratio. In an alternative embodiment of the method, cropping further does not include cropping the radiology sub-images to a standard aspect ratio. In one embodiment of the method, classifying further includes identifying and labeling body parts according to veterinary standard body part labels. In one embodiment of the method, classifying further includes comparing the radiology image to a library of sample standard radiology images.

[0024] An embodiment of the method further includes matching the radiographic image with sample standard images in a library, thereby classifying the radiographic image into one or more body parts. In an embodiment of the method, cropping further includes identifying boundaries within the radiographic image that delineate each classified body part. An embodiment of the method further includes extracting a signature of the radiographic image prior to classifying. In an embodiment of the method, the radiographic image is a radiograph, i.e., from a radiological examination selected from X-ray, magnetic resonance imaging (MRI), magnetic resonance angiography (MRA), computed tomography (CT), fluoroscopy, mammography, nuclear medicine, positron emission tomography (PET), and ultrasound. In an embodiment of the method, the radiographic image is a photograph.

[0025] In one embodiment of the method, the subject is selected from mammals, reptiles, fish, amphibians, chordates, and birds. In one embodiment of the method, the mammal is selected from dogs, cats, rodents, horses, sheep, cattle, goats, camels, alpacas, water buffalo, elephants, and humans. In one embodiment of the method, the subject is selected from pets, livestock, valuable zoo animals, wild animals, and research animals. One embodiment of the method further includes automatically generating at least one report using evaluation of the sub-images by an artificial intelligence processor.

[0026] Aspects of the invention described herein provide a system for analyzing radiological images of a subject, the system including a receiver for receiving radiological images of the subject, at least one processor for automatically executing image identification and processing algorithms to identify, crop, orient, and label at least one body part in the image to obtain sub-images, at least one artificial intelligence processor for evaluating the sub-images, and a device for displaying the sub-images and the evaluated artificial intelligence results.

[0027] In one embodiment of the system, the processor automatically rapid processes the radiographic images to obtain sub-images. In one embodiment of the system, the processor processes the radiographic images to obtain labeled images in less than 1 minute, less than 30 seconds, less than 20 seconds, less than 15 seconds, less than 10 seconds, or less than 5 seconds. One embodiment of the system further includes a library of standard radiographic images. In one embodiment of the system, the standard radiographic images comply with hanging protocols and veterinary standards for body part labels.

[0028] One aspect of the invention described herein provides a method for quickly and automatically preparing radiological images of a subject for display, the method including processing raw radiological images of the subject using a processor to algorithmically classify the images into one or more distinct body part categories by automatically cropping and extracting signatures and comparing the cropped oriented image signatures to a database of signatures of images of known orientations and body parts to obtain the most matching orientation and body part labeling, and presenting each prepared body part-labeled image on a display device for analysis.

[0029] One aspect of the invention described herein provides an improvement in a veterinary radiography diagnostic image analyzer, the improvement including executing a fast algorithm by a processor that pre-processes a radiographic image of a subject to automatically identify one or more body parts within the image, the processor further operative to perform at least one of automatically creating a separate sub-image for each identified body part, cropping and optionally normalizing the aspect ratio of each created sub-image, automatically labeling each sub-image as a body part, and automatically orienting the body parts within the sub-image, the processor further automatically directing the diagnostic sub-images to at least one artificial intelligence processor specialized in evaluating the cropped, oriented, and labeled diagnostic sub-images.

[0030] One aspect of the invention described herein provides a method for identifying and diagnosing the presence of a disease or condition in at least one image of a subject, the method including classifying the image into one or more body parts, labeling and orienting the image to obtain classified, labeled, and oriented sub-images, directing the sub-images to at least one artificial intelligence (AI) processor to obtain evaluation results, comparing the evaluation results and a database having matched pre-written templates or at least one dataset cluster to obtain at least one cluster result, measuring distances between the cluster results and the evaluation results to obtain at least one cluster diagnosis, and assembling the cluster diagnosis to obtain a report, thereby identifying and diagnosing the presence of a disease or condition in the subject. Evaluation result is synonymous with AI result, and AI processor result and classification result are used interchangeably.

[0031] An embodiment of the method further includes acquiring at least one radiological image or one data point of the subject before classifying. An embodiment of the method further includes compiling dataset clusters using a clustering tool selected from the following before comparing: K-means clustering, mean shift clustering, density-based spatial clustering, expectation-maximization (EM) clustering, and agglomerative hierarchical clustering. In an embodiment of the method, compiling further includes acquiring, processing, evaluating, and building a library of the identified and diagnosed datasets and corresponding medical reports selected from radiology reports, laboratory reports, histology reports, physical examination reports, and microbiology reports having the known diseases or conditions. The phrase "medical report" includes any type of medical data. The terms "radiological" and "radiographic" shall have the same meaning.

[0032] In one embodiment of the method, the processing further includes classifying the plurality of identified and diagnosed dataset images into body parts to obtain a plurality of classified dataset images, and further includes orienting and cropping the plurality of classified dataset images to obtain a plurality of oriented, cropped, and labeled dataset sub-images. In one embodiment of the method, the evaluating further includes directing the plurality of oriented, cropped, and labeled dataset sub-images and corresponding medical reports to at least one AI processor to obtain at least one diagnosed AI processor result. In one embodiment of the method, the directing further includes classifying the plurality of oriented, cropped, and labeled dataset sub-images and corresponding medical reports by at least one variable selected from species, breed, weight, sex, and location.

[0033] In one embodiment of the method, building a library of a plurality of identified and diagnosed dataset images further includes creating at least one cluster of diagnosed AI processor results to obtain at least one AI processor sample result, thereby compiling a dataset cluster. In some embodiments, the AI ​​processor sample result is a sample case, sample result, sample point, or sample. These terms are synonymous and used interchangeably. One embodiment of the method further includes assigning at least one cluster diagnosis to the cluster of diagnosed AI processor results. In one embodiment of the method, assigning the cluster diagnosis further includes adding additional information written by a report and / or evaluator within the cluster. In one embodiment of the method, measuring further includes determining a distance between the cluster result and at least one selected from the assessment result, the dataset cluster, and the center of gravity of the cluster result.

[0034] One embodiment of the method further includes selecting results from the case in the cluster with the nearest match, results from another case in the cluster, and centroid cases. In one embodiment of the method, the selecting further includes adding result information of the cluster result by the rater to a report generated from the cluster. One embodiment of the method further includes editing the report by removing portions of the report of cluster diagnoses that have an occurrence rate below a threshold in multiple reports in the cluster. In one embodiment of the method, the report is generated from words, subsentences, sentences, and paragraphs deemed acceptable for use in report generation. Words in the report are taken from the nearest matching exemplar result case. Acceptable words for report generation are excluded if they include at least one identifier selected from subject name, date, reference to previous study, or any other word that may generate a report that is not universally usable for all new cases that most closely match the exemplar result. This selection process is performed using natural language processing (NLP) and language AI.

[0035] In one embodiment of the method, the threshold for the occurrence rate is specified by the evaluator. This threshold can be set between 0.000001% and 99.999999%. In one embodiment of the method, the evaluation results are rapidly processed by a diagnostic AI processor to obtain a report. In one embodiment of the method, the diagnostic AI processor processes the images and obtains a report within an extremely short time interval of less than about 10 minutes, less than about 9 minutes, less than about 8 minutes, less than about 7 minutes, less than about 6 minutes, less than about 5 minutes, less than about 4 minutes, less than about 3 minutes, less than about 2 minutes, or less than about 1 minute. In one embodiment of the method, a library of identified and diagnosed dataset images with known diseases and conditions is classified into at least one of multiple animal species.

[0036] An embodiment of the method further includes identifying the diagnosed AI processor results with an identification tag. An embodiment of the method further includes retaining the original image and the manipulated image, and selecting and adding the AI ​​results and / or subject medical results from the images to a database cluster.

[0037] One aspect of the invention described herein provides a system for diagnosing the presence of a disease or condition in an image and / or medical result of a subject, the system including: a receiver that receives the image and / or medical result of the subject; at least one processor that automatically executes an image identification and processing algorithm to identify, crop, orient, and label at least one body part in the image to obtain a sub-image; at least one artificial intelligence processor that evaluates the sub-image and / or medical result to obtain an evaluation result; and at least one diagnostic artificial intelligence processor that automatically executes a cluster algorithm to compare the evaluation results to obtain a cluster result, measure the distance between the cluster result and a cluster result previously created from a particular dataset defined by one or more variables, the evaluation result to obtain a cluster diagnosis, and assemble a report.

[0038] In one embodiment of the method, the diagnostic AI processor automatically expedites processing of the images and / or medical results to generate a report. In one embodiment of the method, the diagnostic AI processor processes the images and / or medical results to obtain a report in less than about 10 minutes, less than about 9 minutes, less than about 8 minutes, less than about 7 minutes, less than about 6 minutes, less than about 5 minutes, less than about 4 minutes, less than about 3 minutes, less than about 2 minutes, or less than about 1 minute. One embodiment of the method further includes a device for displaying the generated report.

[0039] One aspect of the invention described herein provides a method for diagnosing the presence of a disease or condition in at least one image of a subject, the method including: classifying the image into at least one body part; labeling, cropping, and orienting the image to obtain at least one classified, labeled, cropped, and oriented sub-image; directing the sub-image to at least one artificial intelligence (AI) processor for processing and obtaining evaluation results; comparing the evaluation results to a database library or at least one dataset cluster having a plurality of evaluation results and matched description templates to obtain at least one cluster result; measuring the distance between the cluster results and the evaluation results to obtain at least one cluster diagnosis; assembling the cluster diagnosis and the matched description template to obtain a report; and displaying the report, thereby identifying and diagnosing the presence of the disease or condition in the subject.

[0040] One embodiment of the method further includes analyzing the report after display to confirm the presence of the disease or condition. An alternative embodiment of the method further includes editing the pre-written template. In one embodiment of the method, obtaining the report has a process time of less than about 5 minutes, less than about 2 minutes, or less than about 1 minute. In one embodiment of the method, obtaining the report has a process time of less than about 10 minutes, less than about 7 minutes, or less than about 6 minutes.

[0041] In one embodiment of the method, processing the sub-images further includes training an AI processor to diagnose the presence of a disease or condition in the image of the subject. In one embodiment of the method, training the AI ​​processor further includes communicating the library of training images to the AI ​​processor to create an AI model, storing the AI ​​model in a database, testing the AI ​​model with expected positive and negative data not used for training, and comparing the actual test set results with the expected test set results.

[0042] In one embodiment of the method, the library of training images includes positive control training images and negative control training images. In one embodiment of the method, the positive control training images have the disease or condition of the training images. In one embodiment of the method, the negative control training images do not have the disease or condition of the training images. In various embodiments of the method, the negative control training images may have a disease or condition other than the disease or condition of the training images. In one embodiment of the method, the library of training images further includes at least one of medical data, metadata, and auxiliary data.

[0043] One aspect of the invention herein describes a novel system having several analysis stages, including identifying whether a received image contains a specific body part in a specific orientation (e.g., a side view), appropriately cropping the image, and evaluating the cropped image by comparing the image to a target AI model. In various embodiments, newly received images are preprocessed to automatically identify and label one or more body parts and / or views represented in the image without user input or intervention. In some embodiments, the image is automatically cropped to generate one or more sub-images corresponding to each identified body part / view. In some embodiments, the image and / or sub-images are selectively processed to a target AI processor configured to evaluate the identified body part / view, to the exclusion of the remainder of the AI ​​processors in the system.

[0044] In some embodiments, the radiology image preprocessor 106 additionally or alternatively tags the entire image 102 to identify body parts and / or views identified within the image 102, and then passes the entire image 102 only to the AI ​​processor 104 that corresponds to the applied tags. Thus, in such embodiments, the AI ​​processor 104 is responsible for cropping the image 102 to focus on those parts of interest for further analysis using one or more trained machine learning models or otherwise. In some embodiments, in addition to tagging the image 102 as corresponding to a particular body part / view, the radiology image preprocessor 106 also crops the image 102 to primarily focus on areas of the image that actually represent parts of the animal and to remove, as much as possible, black borders around those areas. In some embodiments, performing such cropping facilitates further cropping and / or other processing by the AI ​​processor 104 that is later deployed to evaluate the particular body part / view that corresponds to the applied tag.

[0045] The radiographic image preprocessor 106 may be implemented in any of several ways. In some embodiments, for example, the radiographic image preprocessor 106 employs one or more algorithms to identify one or more features indicative of one or more particular body parts and automatically crop the image 102 to focus on regions containing such features and / or regions that actually represent an animal. In some implementations, such algorithms are implemented using elements of the OpenCV-Python library, for example. A description of the Open Source Computer Vision ("OpenCV") library, as well as related documentation and tutorials, can be found using OpenCV's uniform resource locator (URL). The entire contents of the materials accessible via the URL are incorporated herein by reference. In some embodiments, the radiology image preprocessor 106 additionally or alternatively employs image matching techniques to compare the image 102 and / or one or more of its cropped sub-images 108 to a repository of stored images known to represent particular views of particular body parts, and the image 102 and / or sub-image 108 is identified as representing the body part / view that is found to have the strongest correlation with one or more of the stored images. In some embodiments, an AI processor trained to perform body part / view identification is additionally or alternatively employed within the radiology image preprocessor 106.

[0046] In some embodiments, one or more of the AI ​​processors described herein are implemented using the TensorFlow platform. Descriptions, documentation, and tutorials for the TensorFlow platform can be found at the TensorFlow website. The entire contents of the materials accessible via the website are incorporated herein by reference. The TensorFlow platform and methods for building AI processors are fully described in Hope, Tom et al., Learning TensorFlow: A Guide to Building Deep Learning Systems. O'Reilly., 2017, which is incorporated herein by reference in its entirety.

[0047] In the example shown in FIG. 3 , the pre-processing performed by the radiology image preprocessor 106 includes: (1) an optional “general” auto-cropping step (reflected in the first five log entries delineated by bracket 306) according to which the image 302 is first cropped to focus on regions of the image primarily representing parts of the animal and to remove as much of the black border around those regions as possible; (2) a “classified” auto-cropping step (reflected in log entries 6-9 within bracket 306) according to which an initial effort is made, e.g., using elements of the OpenCV-Python library, to identify a particular body part / view and crop the image 302 to focus thereon; and (3) an AI region labeling or “image matching” step (reflected in the last three log entries delineated by bracket 306) according to which the image 302 and / or one or more of its cropped sub-images 304 a-b are compared to a repository of stored images known to represent particular views of a particular body part. As shown by the corresponding timestamps, the general auto-cropping step was observed to complete in 2 seconds, the classified auto-cropping step was observed to complete in 3 seconds, and the image matching step was observed to complete in 19 seconds.

[0048] The time taken by thoracic lateral AI processor 104a to determine whether image 302 included a lateral view of the animal's thorax was 4 seconds, as indicated by the log entry depicted by bracket 308a in Figure 3. Similarly, abdominal lateral AI processor 104c took 4 seconds to determine whether image 302 included a lateral view of the animal's abdomen, as indicated by the log entry depicted by bracket 308b.

[0049] If the system had instead had to process the newly received image 302 with all of the possible AI processors 104a-1 rather than just the two corresponding to the body part / view identified by the radiology image preprocessor 106, the time taken by the AI ​​processors to complete the analysis would have been significantly longer and / or consumed significantly more processing resources. In a system including 30 different AI processors 104, for example, the process of simply identifying the relevant AI model for identifying the state of the imaged animal would have required at least 120 seconds of processing time by the AI ​​processors 104 (i.e., 30 AI processors at 4 seconds per processor), and likely much longer if multiple possible orientations of the image were considered by each of the AI ​​processors 104. On the other hand, by employing the radiology image preprocessor 106, the identification of the relevant AI model was observed to require only 8 seconds of processing time by the AI ​​processors 104 plus 24 seconds of preprocessing time by the radiology image preprocessor 106.

[0050] It is useful to process radiological images of animals using artificial intelligence (AI) processors, e.g., trained neural networks, to determine the probability that the imaged animal has a particular pathology. Typically, separate AI processors are used to evaluate each body part (e.g., chest, abdomen, shoulders, forelimbs, hindlimbs, etc.) and / or a particular orientation of each such body part (e.g., ventrodorsal (VD) view, lateral view, etc.), with each such AI processor determining, for each body part and / or orientation, the probability that a particular condition exists for the body part in question. Each such AI processor may include multiple trained models that evaluate each condition or organ within the imaged region. For example, for a lateral view of an animal's chest, the AI ​​processor may employ different models to determine the probability that the animal has a particular lung-related condition (e.g., perihilar infiltrates, pneumonia, bronchitis, pulmonary nodules, etc.).

[0051] Detection of a single medical condition, such as the presence or absence of pneumonia or pneumothorax, is currently performed in radiology AI. In contrast to single-disease detection by current radiology AI, human radiologists analyze radiographs with a holistic approach by simultaneously assessing the presence or absence of many conditions. A limitation of current AI processes is the need to use separate AI detectors for each specific condition. However, a combination of conditions can result in a diagnosis of a broader disease. For example, in some cases, one or more diagnoses obtained from a radiograph are caused by several broader diseases. Identifying a broader disease present in a subject's radiograph requires the use of supplementary diagnostic results in a process known as differential diagnosis. These supplementary diagnostic results are extracted from blood tests, the patient's medical history, biopsies, or other tests and processes in addition to the radiograph. Current AI processes focus on a single diagnostic result and are unable to identify broader diseases that require differential diagnosis. Described herein is a novel AI process that can combine multiple diagnostic results to diagnose a broader disease.

[0052] AI processes currently use limited radiographic images directed at specific regions, as is typical in radiological imaging of human subjects. In contrast, veterinary radiology typically includes multiple body parts within a single radiograph. A novel AI evaluation process is described herein that evaluates all body parts included in the study, providing a broader evaluation than is expected in veterinary radiology.

[0053] The current conventional workflow for AI reporting of a single disease process is shown in Figure 4. The conventional single-state report shown in Figure 4 is insufficient for differential diagnosis of radiographs. Furthermore, using individualized rules for each combination of assessment results is inefficient for generating reports and fails to meet the reporting standards expected of veterinary radiologists. Even for a single disease process, identifying specific conditions—e.g., normal, minimal, mild, moderate, and severe severity—results in an exponential number of AI model result templates. The process of creating and selecting a single report template from aggregate AI model diagnostic results scales exponentially with the number of AI models. The number of AI models for a single disease process results in 57 different templates for five different severity levels, as shown in Figures 5A–5N. Therefore, manually generated reports for each combination of AI model diagnostic results do not scale well to a large number of AI models being interpreted together.

[0054] Automated system for AI analysis Described herein is a novel system for analyzing images of a subject animal, the system including a receiver for receiving an image of the subject; at least one sub-image processor for automatically identifying, cropping, orienting, and labeling at least one body part in the image to obtain sub-images; at least one artificial intelligence evaluation processor for evaluating the sub-images for the presence of at least one condition; at least one synthesis processor for generating an overall result report from the at least one sub-image evaluation and optionally non-image data; and a device for displaying the sub-images and the overall composite diagnostic result report.

[0055] The system provides substantial advances in veterinary diagnostic image analysis by (1) automating sub-image extraction using a sub-image processor, a task typically performed manually or with user assistance, and (2) using a synthesis processor to synthesize large collections of evaluation results and other non-image data points into concise, cohesive, and comprehensive reports.

[0056] A case includes a collection of one or more images of a subject animal and may include non-image data points such as, but not limited to, age, sex, location, medical history, and other medical test results. In one embodiment of the system, each image is sent to multiple sub-image processors that generate many sub-images of different views of multiple body parts. Each sub-image is processed by multiple assessment processors to generate multiple assessment results for many different conditions, findings, or other characteristics across the many body parts. A synthesis processor processes all or a subset of the assessment results and non-image data points to generate an integrated composite diagnostic result report. In one embodiment of the system, multiple synthesis processors generate multiple composite diagnostic result reports from different subsets of the assessment results and non-image data points. These diagnostic reports are assembled with auxiliary data to create a final integrated composite diagnostic result report.

[0057] In one embodiment of the system, each synthesis processor operates on a subset of sub-images and non-image data points corresponding to a body part, e.g., the thorax or the abdomen. Each synthesis diagnostic report includes the body part, as is typical practice in veterinary radiology. The overall synthesis diagnostic results report includes descriptive data of the subject, e.g., name, age, address, breed, and multiple sections corresponding to the output of each synthesis processor, e.g., a thorax results section and an abdominal results section.

[0058] In one embodiment of the system, the subject is selected from mammals, reptiles, fish, amphibians, chordates, and birds. Mammals are dogs, cats, rodents, horses, sheep, cattle, goats, camels, alpacas, water buffalo, elephants, and humans. Subjects are pets, livestock, valuable zoo animals, wild animals, and research animals.

[0059] The images received by the system are images from radiological examinations such as X-rays (radiographs), magnetic resonance imaging (MRI), magnetic resonance angiography (MRA), computed tomography (CT), fluoroscopy, mammography, nuclear medicine, positron emission tomography (PET), and ultrasound. In some embodiments, the images are photographs.

[0060] In some embodiments of the system, analyzing the images of the subject generates and displays a comprehensive composite results report in less than about 20 minutes, less than about 10 minutes, less than about 5 minutes, less than about 1 minute, less than about 30 seconds, less than about 20 seconds, less than about 15 seconds, less than about 10 seconds, or less than about 5 seconds.

[0061] Sub Image Processor The sub-image processor orients, crops, and labels at least one body part in the image to automatically and quickly acquire sub-images. The sub-image processor orients the image by rotating the image to a standard orientation for a particular view. The orientation is specified by a veterinary radiography standard hanging protocol. The sub-image processor crops the image by identifying boundaries in the image depicting one or more body parts and creating a sub-image that includes image data within the identified boundaries.

[0062] In some embodiments, the boundary is a uniform aspect ratio. In alternative embodiments, the boundary is not a uniform aspect ratio. The sub-image processor labels the sub-images by reporting the boundary and / or location of each body part contained within the sub-image. Body parts may be, for example, thorax, abdomen, spine, forelimbs, left shoulder, head, neck, etc. In some embodiments, the sub-image processor labels the sub-images according to veterinary radiology standard body part labels.

[0063] The sub-image processor orients, crops, and labels one or more sub-images by matching the image to a plurality of reference images in at least one of a plurality of libraries, each of which contains a respective plurality of reference images that may be species-specific or non-specific.

[0064] The sub-image processor extracts a signature of the image before orienting, cropping, and / or labeling the image, thereby enabling rapid matching of the image or sub-image to a similar reference image. The sub-image processor processes the image to obtain a sub-image in less than about 20 minutes, less than 10 minutes, less than about 5 minutes, less than about 1 minute, less than about 30 seconds, less than about 20 seconds, less than about 15 seconds, less than about 10 seconds, or less than about 5 seconds.

[0065] Evaluation Processor An artificial intelligence assessment processor assesses the sub-images for the presence or absence of a condition, finding, or other feature. The assessment processor reports a probability of the presence of the condition, finding, or feature.

[0066] The evaluation processor diagnoses a medical condition from the sub-images. The evaluation processor evaluates the sub-images for non-medical features, such as proper positioning of the subject. The evaluation processor generates instructions to correct the positioning of the subject.

[0067] Typically, assessment processor training includes negative control / normal and positive control / abnormal training sets for a condition, finding, or other characteristic. The positive control / abnormal training set typically includes cases assessed for the presence of the condition, finding, or other characteristic. The negative control / normal training set includes cases assessed for the absence of the condition, finding, or other characteristic and / or cases that are considered completely normal. In some embodiments, the negative control / normal training set includes cases assessed for the presence of other conditions, findings, or characteristics different from those of interest. Thus, the assessment processor is robust.

[0068] The assessment processor processes the sub-images and reports the presence of the condition in less than about 20 minutes, less than about 10 minutes, less than about 5 minutes, less than about 1 minute, less than about 30 seconds, less than about 20 seconds, less than about 15 seconds, less than about 10 seconds, or less than about 5 seconds.

[0069] Synthesis Processor The synthesis processor receives at least one evaluation from the evaluation processor and generates a comprehensive results report. The synthesis processor may include non-image data points, such as species, breed, age, weight, location, sex, medical test history including blood, urine, and fecal tests, radiology reports, laboratory reports, histology reports, physical examination reports, microbiology reports, or other medical and non-medical tests or results. The subject's case specimen results include at least one image, the results of the associated evaluation processor, and a collection of zero or more recent non-image data points.

[0070] In one embodiment of the method, the synthesis processor uses the case sample results to select pre-written templates to output as a comprehensive results report.

[0071] The template is automatically customized based on the case sample result elements to provide a customized overall results report.

[0072] The synthesis processor assigns case sample results for a subject to cluster groups. Cluster groups include other similar case sample results from a reference library of case sample results from other subjects. In some cases, cluster groups include partial case sample results, such as result reports. The reference library includes case sample results with known diseases and conditions from at least one of multiple animal species. New case sample results are added to the reference library to improve the performance of the synthesis processor over time. The synthesis processor assigns coordinates representing the location of each case sample result within the cluster group.

[0073] A single overall result report is assigned to the entire cluster group, and the overall result report is assigned to the subject by the synthesis processor. In some embodiments, several overall result reports are assigned to various case sample results within the cluster and / or various custom coordinates within the cluster, such as cluster centroids, where there are no associated case sample results. The coordinates of a subject's case sample result are used to calculate the distance to the nearest or non-nearest case sample result or custom coordinate with an associated overall result report, and then assigned to the subject.

[0074] The Comprehensive Results Report is written by an expert human reviewer. In an alternative embodiment, one or more Comprehensive Results Reports are generated from existing radiology reports. The existing radiology reports are modified using natural language processing (NLP) to remove content that is not universally applicable, such as names, dates, references to previous studies, etc., to create a suitable Comprehensive Results Report. Statements included in the Comprehensive Results Report are removed or edited if the statement does not meet a threshold for occurrence within a cluster.

[0075] The synthesis processor outputs a composite result report assigned to the subject, thereby identifying and diagnosing the presence of one or more findings, diseases, and / or conditions in the subject. Cluster groups are established from a reference library of case sample results using a clustering tool selected from the following: K-means clustering, mean shift clustering, density-based spatial clustering, expectation-maximization (EM) clustering, and agglomerative hierarchical clustering.

[0076] The synthesis processor processes the case sample results and generates a comprehensive results report in less than about 20 minutes, less than about 10 minutes, less than about 9 minutes, less than about 8 minutes, less than about 7 minutes, less than about 6 minutes, less than about 5 minutes, less than about 4 minutes, less than about 3 minutes, less than about 2 minutes, less than about 1 minute, less than about 30 seconds, less than about 20 seconds, less than about 15 seconds, less than about 10 seconds, or less than about 5 seconds.

[0077] Clustering is an AI technique for grouping unlabeled examples by the similarity of each example's features. A process for clustering patient studies based on AI processor diagnostic results, as well as non-radiological and / or non-AI diagnostic results, is described herein. The clustering process groups reports that share similar diagnoses or output reports, thereby facilitating the overall detection of conditions or broader diseases in a scalable manner.

[0078] Described herein are novel systems and methods with multiple analysis stages that combine multiple methods of AI predictive image analysis against a radiographic image and report library database with newly received image evaluation to accurately diagnose and report radiology cases. In various embodiments, the novel systems described herein automatically detect the field of view and area, or area covered by each radiology image.

[0079] In some embodiments, the system preprocesses newly received radiological images 102 using a radiological image preprocessor 106 to crop, rotate, flip, create sub-images, and / or normalize image exposure prior to AI evaluation. If more than one body region or view is identified, the system further crops the image 102 to generate one or more sub-images 108a, 108b, and 108c corresponding to each identified region and view. In some embodiments, the system selectively processes and directs images and / or sub-images to targeted AI processors configured to evaluate the identified regions / views. Image 108a is directed only to AI processor 104a, which is the thoracic lateral AI processor. Image 108b is directed only to AI processor 104c, which is the abdominal lateral AI processor. Image 108c is directed only to AI processor 104k, which is the pelvic lateral AI processor. Images are not directed to untargeted AI processors or to the remaining AI processors in the system. For example, chest image Figure 7 is directed to one or more AI processors for diseases listed in Figure 6, such as heart failure, pneumonia, bronchitis, interstitial, diseased lung, hypoplastic trachea, cardiomegaly, pulmonary nodules, pleural effusion, gastritis, esophagitis, bronchiectasis, pulmonary hyperinflation, pulmonary vascular hypertrophy, thoracic lymphadenopathy, etc.

[0080] In some embodiments, the AI ​​model processor is a binary processor that provides a binary result of normal or abnormal. In various embodiments, the AI ​​model processor provides a diagnosis of normal or abnormal along with an identification of the severity of the particular condition, which may be classified as, for example, normal, minimal, mild, moderate, or severe.

[0081] In some embodiments, newly received AI model processor results are displayed on a user interface. See FIGS. 9A-9E. Average AI model processor results for each model are collected and displayed from the individual image or sub-image evaluation results. See FIG. 9A. The user interface displays the individual image or sub-image and the AI ​​model processor results for that image. See FIGS. 9B-9E. The AI ​​analysis is completed in less than 1, 2, or 3 minutes.

[0082] In some embodiments, one or more clusters are constructed by the system using AI processor diagnosis results from a library of known radiological images and corresponding radiology report databases to develop nearest-match cases or AI processor "exemplar results" for one or more AI processor results. An exemplar result includes at least one image, a set of associated assessment processor results, and a set of zero or more non-image data points, such as age, sex, location, breed, and medical test results. The synthesis processor assigns coordinates representing the location of each case exemplar result within a cluster group. Thus, if two cases have similar exemplar results, the diagnosis is similar or nearly identical, and a single overall result report applies to the two cases. In some embodiments, a single exemplar result is assigned to an entire cluster, and the exemplar result is assigned to the subject case located within the cluster. In some embodiments, multiple exemplar results are assigned to a cluster that is either linked to a specific coordinate (e.g., centroid) within the cluster or to a specific dataset case. In some embodiments, the exemplar results are either manually written or automatically generated from existing radiology reports linked to the cases.

[0083] In some embodiments, the user specifies various parameters for creating a cluster from a library of known radiological images and corresponding radiology report database through the user interface of FIG. 11 . The user assigns a name 1101 for the new cluster under “Code.” The user selects various parameters to create the cluster. The user selects cases by selecting a case start date 1102 and a case end date 1103. The user selects cases by selecting a case start ID 1104 and a case end ID 1105. The user selects the maximum number of cases 1106 to be included in the cluster. The user selects the species 1107, such as canine, feline, canine or feline, human, avian, pet, livestock, etc., for the cases to be included in the cluster. The user selects specific diagnostic modalities 1108, such as X-ray, CT, MRI, blood analysis, urinalysis, etc., to be included in the creation of the cluster.

[0084] In various embodiments, the user specifies that the evaluation results be separated into a specific number of clusters. The number of clusters ranges from a minimum of one cluster to a maximum of clusters limited only by the total number of cases input into the cluster. The system uses non-radiological and / or non-AI diagnostic results, such as blood tests, patient history, or other tests or processes, in addition to AI processor diagnostic results, to construct one or more clusters. The clusters are listed in numerical format in a comma-separated value (CSV) file format, as shown in FIGS. 12A-12B. The CSV file lists the case IDs 1201 of the cases in the cluster. The average evaluation result 1202 of each binary model for a particular case ID is listed in the CSV file. The cluster label or cluster location 1203 containing a particular case based on the set of evaluation results is listed in the CSV file. The CSV file lists the cluster coordinates. The case ID 1204 of the centroid or center of a particular cluster is listed in the CSV file.

[0085] In various embodiments, the clusters are represented by a clustering graph. See Figure 13. The clustering graph is created by dividing the average rating results into different clusters according to user-defined parameters 1102-1108. The different clusters are represented by sets of points plotted on the graph. The clustering graph in Figure 13 shows 180 clusters of various sizes.

[0086] In some embodiments, the user interface shows the AI ​​cluster model that is generated based on user-defined parameters 1102-1108. See FIG. 14. The user interface shows the screening evaluation configuration in which the user assigns a particular "cluster model" 1502 to a particular "screening evaluation configuration" name 1501. The screening evaluation configuration status 1503 provides additional information about the configuration, such as whether the configuration is in live, test, or draft mode. Live mode is for production, and test or draft mode is for development.

[0087] In some embodiments, the user interface details a particular cluster model 1601 (chest 97). See FIG. 16A. In some embodiments, the user interface lists the AI ​​assessment classifier types 1602 included in the cluster. The user interface displays additional parameters used to build the cluster, such as a species 1603 specific to the cluster model, a maximum number of cases with assessment results 1604, or start and end dates 1605 of the cases used to create the cluster. The user interface provides a link 1606 to a comma-separated values ​​(CSV) file that displays the cluster in a numeric table format. The user interface lists subclusters 1608 created from parameters 1602-1605. The user interface displays the total number of subclusters 1609 created for the cluster group. The user interface provides a centroid case ID 1610 for each subcluster. A log for building the cluster is provided in the user interface. See FIG. 16B.

[0088] In various embodiments, the system utilizes one or more AI processors to evaluate newly received non-diagnostic images to obtain newly received evaluation results, which the system compares to one or more clusters obtained from a library of known radiological images and corresponding radiology report databases.

[0089] The user imports the newly received AI processor results into the AI ​​evaluation tester (see Figure 17A). The user specifies the screening evaluation type 1702 and the corresponding cluster model 1703.

[0090] The system compares the newly received evaluation results, as well as non-radiological and / or non-AI diagnostic results, to one or more clusters obtained from a library of known radiological images and corresponding radiology report databases, among other available results. The system measures the distance between the location of the newly received AI processor results and the cluster results, and generates a radiologist report using one or more cluster results. In some embodiments, the system selects to use the entire radiologist report or a portion of the radiologist report from a known cluster result depending on the location of the newly received AI processor result relative to the known cluster results. In various embodiments, the system chooses to use the entire radiologist report or a portion of the radiologist report from other results in the same cluster. In some embodiments, the system selects to use the entire radiologist report or a portion of the radiologist report from the centroid of the cluster result.

[0091] The user interface displays the results of the AI ​​evaluation tester. See FIG. 18A. In various embodiments, the diagnosis and conclusion findings 1801 from the radiologist report that most closely matches the evaluation result based on the previously created cluster results are displayed. In some embodiments, evaluation findings 1802 are selected from the radiologist reports within the cluster of evaluation results and filtered based on the occurrence of specific sentences in the findings section of the particular cluster. In some embodiments, recommendations 1803 from the radiologist reports within the cluster are selected based on the occurrence of each sentence or similar sentences in the recommendations section of the cluster. The user interface displays the radiologist report 1804 for the cluster and the radiologist report 1805 for the centroid of the cluster. The user interface allows the user to edit the report by adding or deleting specific sentences to the findings section 1809, conclusion section 1810, or recommendation section 1811. See FIG. 18D. The radiologist reports for the nearest match database cases are used to generate a radiology report for the newly received radiology images. The sentences in the radiology report based on a particular cluster result are ranked and listed according to rank and frequency of occurrence, see Figures 18A and 18B.

[0092] In various embodiments, the system utilizes one or more AI processors to evaluate newly received non-diagnostic images to obtain newly received evaluation results, which the system compares to one or more clusters obtained from a library of known radiological images and corresponding radiology report databases.

[0093] The system compares the newly received evaluation results, as well as non-radiological and / or non-AI diagnostic results, to one or more clusters obtained from a library of known radiological images and corresponding radiology report databases, among other available results. The system measures the distance between the location of the newly received AI processor results and the cluster results, and generates a radiologist report using one or more cluster results. In some embodiments, the system selects to use the entire radiologist report or a portion of the radiologist report from a known cluster result depending on the location of the newly received AI processor result relative to the known cluster results. In various embodiments, the system chooses to use the entire radiologist report or a portion of the radiologist report from other results in the same cluster. In some embodiments, the system selects to use the entire radiologist report or a portion of the radiologist report from the centroid of the cluster result.

[0094] In some embodiments, one or more of the AI ​​processors described herein are implemented using the TensorFlow platform. A description of the TensorFlow platform, as well as documentation and tutorials related thereto, can be found on the TensorFlow website. The entire contents of the materials accessible at the TensorFlow website are incorporated herein by reference in their entirety.

[0095] In some embodiments, one or more of the clustering models described herein are implemented using the Plotly platform. A description of the Plotly platform, as well as documentation and tutorials related thereto, can be found on the scikit-learn website. The entire contents of the materials accessible on the scikit-learn website are incorporated herein by reference in their entirety. Methods for developing AI processors and clustering models using the TensorFlow platform and Scikit-learn are fully described in the following references: Geron Aurelien. Hands-on Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. O'Reilly, 2019; Hope, Tom, et al. Learning TensorFlow: A Guide to Building Deep Learning Systems. O'Reilly, 2017; and Sievert, Carson. Interactive Web-Based Data Visualization with R, Plotly, and Shiny. CRC Press, 2020. Each of these references is incorporated herein by reference in its entirety.

[0096] If a radiologist were to evaluate each model and attempt to create rules based on the separate AI processor results found together, rule and report creation would take an enormous amount of time. Additionally, adding a single additional AI processor model to this scenario becomes exponentially more difficult as the number of AI processor models already incorporated increases. By employing a new workflow of AI processor result clustering or "exemplar result" comparison between new images and known datasets to create radiologist reports, the problem of manual report construction when multiple AI processor results are found is solved. Manually constructing reports via individual AI processor results and rule creation previously took months, but the new workflow now takes minimal time.

[0097] In some embodiments of the system, the components used for AI evaluation are as illustrated in Figure 21 A. In various embodiments of the system, the components used for image matching AI processing are as illustrated in Figure 21 B.

[0098] In various embodiments of the system, a server architecture for AI radiology image analysis includes pre-processing radiology images, analyzing the images using an AI diagnostic processor, and generating reports based on the clustering results. See Figure 22B. In some embodiments, various servers are used, including an NGINX load balancing server 2202, a V2 cloud platform 2203, a database Microsoft SQL server 2207, and a data store server 2208.

[0099] In various embodiments of the system, a user flags cases for training the AI ​​system. In some embodiments of the system, a user flags cases when the radiology report needs editing because it is inaccurate, or when the report is insufficient, or when the case has a new diagnosis and therefore the radiology report needs new language for the diagnosis.

[0100] A simplified sequence of the AI ​​autocropping and evaluation workflow is shown in Figures 23A-23T. A user accesses the V2 end-user application 2301 to upload images (in image formats such as DICOM, JPEG, JPG, PNG, etc.) to be analyzed by the system. In some embodiments, the images are uploaded 2305 directly to the VetImages application. V2 processes 2302 the images, saves them to a data store, and requests 2303 VetImages to further process the images. VetImages receives the request from V2 and begins asynchronous processing 2304. VetImages accesses 2307 the images from the data store and requests 2308 VetConsole to preprocess the images. VetConsole uses OpenCV 2309 to improve the image quality and autocrop 2310 the images. Tasks after accessing the images from the data store are performed by the sub-image processor.

[0101] VetConsole sends the improved quality auto-cropped image to VetImages, which requests VetConsole to analyze the image and classify 2311 the chest, abdomen, and pelvis in the image. VetConsole classifies 2312 the chest, abdomen, and pelvis in the image and sends the coordinates to VetImages. VetImages sends 2313 the image and coordinates to ImageMatch verification, which matches the image and coordinates with correctly classified images in its database and sends 2314 the matched image distance and path to VetImages. The VetImages application receives the matched image data and uses database information to identify the body part 2315. The next task is to determine the image orientation. The image is rotated and flipped 2317. After each rotation and flip, the image is sent 2318 to the ImageMatch orientation application, which compares it to the matched image and measures the distance and image path between the matched image and the newly received image. The ImageMatch orientation application sends 2319 a result with the distance and image path between the newly received image and the matched image. The orientation of the newly received image with the smallest distance from the matched image is selected 2320 by the VetImages application. The process of checking each orientation and each flip is repeated until the image has been rotated 360° and flipped at the appropriate angle. In some embodiments, the image with the selected orientation is sent to VetAI to detect the chest 2321 and abdomen 2323 and obtain coordinates for cropping the image to obtain a sub-image with the chest 2322 and abdomen 2324. The process of obtaining the coordinates works with TensorFlow.

[0102] The VetImages application obtains coordinates from ImageMatch verification and crops 2325 the image according to the coordinates to obtain a sub-image. The sub-image is sent 2326 to the ImageMatch verification application for matching. The database image is matched 2327 with the sub-image, and the distance and image path between the matched database image and sub-image are sent to the VetImages application. The VetImages application receives 2328 the distance and image path data and uses the data received from the matched image to identify the body part. The VetImages application analyzes each sub-image to check if it is valid. If the sub-image is not valid 2331, a generic cropped image from the VetConsole application is saved 2332 to a database or data store. If the sub-image is valid 2330, the sub-image is saved 2332 to a database or data store. The image saved to the database or data store is the cropped image used for further processing or analysis. The VetImages application stores 2332 data obtained from the VetConsole for sub-images or full crop images in a database.

[0103] The following tasks are performed by the evaluation processor: The VetImages application sends the cropped images to the VetAI application for positioning evaluation 2333. Data received by the VetImages application from the VetAI application 2334 is stored in a database 2335 and a signal is sent to the V2 application to send an email 2336 to the clinic. The VetImages application accesses 2337 a live AI model from the database. The cropped images are sent 2339 to the appropriate AI model in the VetAI application based on the body part of the cropped image. The appropriate AI model is pre-specified for each body part. The VetAI application sends the AI ​​evaluation labels and machine learning (ML) AI evaluation results 2340 to the VetImages application, which stores 2341 these data in a database for the cropped images. The VetImages application calculates 2342 a label and probability for the image based on the AI ​​evaluation results of the cropped images. The process of sending 2339 the cropped image for AI evaluation results is repeated 2342 until a given AI model is processed 2338 .

[0104] The VetImages application analyzes 2344 whether each image from the case is processed to obtain the results of the AI ​​assessment. If all images from the case have not been processed, VetImages returns to process the next image in the case. If all images from the case have been processed, VetImages calculates 2345 the label and probability for the entire case based on the label and probability of each cropped image. The VetImages application then changes 2346 its status to live and screening assessment type from the database. Changing the status of the VetImages application to live causes tasks to be performed by the synthesis processor. The VetImages application evaluates 2347 whether all screening assessments have been completed. If not, the VetImages application evaluates 2348 whether the screening assessments should be completed by clustering. If the screening assessment should be completed by clustering, the AI ​​assessment results for the processed images are sent 2349 to the VetAI application and the best match cluster result is sent 2350 to the VetImages application, which generates screening results based on the best match cluster result and stores them in a database 2351. If the VetImages application determines that the screening assessment should not be performed using clustering, the discovery rules are accessed 2352 and the AI ​​assessment results are processed based on the discovery rules to obtain screening results and store them in a database 2353. The process of obtaining screening results and storing them in a database is repeated until the screening assessment for all images in the case is completed and a complete results report is obtained 2354.

[0105] The VetImages application evaluates 2355 whether the subject's species has been identified and stored in a database. If the species has not been identified, the VetAI application evaluates 2357 the subject's species and sends the species evaluation results to the VetImages application. The species evaluation tasks 2356-2357 are performed by an evaluation processor. In some embodiments, the VetImages application evaluates 2358 whether the species is Canidae. If the species is positively identified, for example, as Canidae, the case is flagged 2359 and the evaluation is attached to the results report. The VetImages application notifies 2360 V2 that the case evaluation is complete. The V2 application evaluates 2361 whether the case is flagged. If the report is flagged, the results report is saved 2362 to the case documentation and the results report is emailed 2363 to the client. If the report is not flagged, the results report is emailed 2363 to the client without saving the report to the case document.

[0106] Clustering Clustering is a type of unsupervised learning method in which references are derived from a dataset consisting of labeled, unresponsive input data. In general, clustering is used as a process to find meaningful structures, underlying processes that explain, generative features, and groupings inherent in a set of examples.

[0107] Clustering is the task of dividing a population or data points into groups such that data points within the same group are similar to other data points within the same group and dissimilar to data points in other groups. Thus, clustering is a method of collecting objects into groups based on similarities and dissimilarities between the objects.

[0108] Clustering is an important process for identifying inherent groupings among unlabeled data. There are no standards for good clustering, as clustering depends on the user choosing criteria that are useful to meet the user's goals. For example, clusters can be based on finding representatives of homogeneous groups (data reduction), finding "natural clusters" and describing their unknown properties ("natural" data types), finding useful and relevant groupings ("useful" data classes), or finding unusual data objects (outlier detection). The algorithm makes assumptions about what constitutes similarity between points, and each assumption creates different but equally valid clusters.

[0109] Clustering method: There are various methods for clustering, such as:

[0110] Density-based methods: These methods consider clusters as dense regions that have a certain degree of similarity and are distinct from less dense regions of space. These methods have good accuracy and the ability to merge two clusters. Examples of density-based methods are Density-Based Spatial Clustering of Applications with Noise (DBSCAN), Ordering Points to Identify Clustering Structure (OPTICS), etc.

[0111] Hierarchy-based methods: Clusters formed by this method form a tree-like structure based on a hierarchy. New clusters are formed using previously formed clusters. Hierarchy-based methods are divided into two categories: agglomerative (bottom-up approach) and divisive (top-down approach). Examples of hierarchy-based methods are Clustering Using Representatives (CURE), Balanced Iterative Reducing Clustering, Hierarchies (BIRCH), etc.

[0112] Partitioning methods: Partitioning methods divide objects into k clusters, with each partition forming one cluster. This method is used to optimize an objective criteria similarity function. Examples of partitioning methods are K-means, Clustering Large Applications based on Randomized Search (CLARANS), etc.

[0113] Grid-based methods: In grid-based methods, the data space is formulated into a finite number of cells that form a grid-like structure. All clustering operations performed on these grids are fast and independent of the number of data objects. Examples of grid-based methods are Statistical Information Grid (STING), Wave Cluster, Clustering In Quest (CLIQUE), etc.

[0114] K-means clustering K-means clustering is an unsupervised machine learning algorithm. Typically, unsupervised algorithms make inferences from a dataset using only input vectors, without reference to known or labeled outcomes. The goal of K-means is to simply group similar data points and discover underlying patterns. To achieve this goal, K-means looks for a fixed number (k) of clusters in the dataset.

[0115] A cluster refers to a set of data points that are aggregated together due to a certain similarity. The target number k refers to the number of centroids the user desires in the dataset. A centroid is a virtual or real location that represents the center of a cluster. Every data point is assigned to each of the clusters by reducing the sum of squares within the cluster. The K-means algorithm identifies k centroids and then assigns every data point to the nearest cluster while keeping the centroids as small as possible.

[0116] The "mean" in K-means refers to the averaging of data to find the centroid. To process the training data, the K-means algorithm in data mining starts with a first group of randomly selected centroids that are used as the starting point for each cluster, and then performs iterative calculations to optimize the centroid positions. The algorithm stops creating and optimizing clusters when the centroids stabilize and their values ​​remain unchanged, either because clustering is successful or the defined number of iterations has been achieved.

[0117] The method for K-means clustering follows a simple approach: classifying a given dataset through a certain number of clusters (assuming k clusters), which is fixed a priori. K centers, or centroids, are defined, one for each cluster. The next step is to take each point belonging to the given dataset and associate it with its nearest center. If there are no pending points, the first step is completed, and early group ageing is performed. The next step is to recalculate k new centroids as the centroids of the clusters resulting from the previous step. After calculating the k new centroids, a new connection is performed between the same dataset point and the nearest new center, thereby creating a loop. As a result of this loop, the k centers gradually change their positions until they stop changing their positions. The K-means clustering algorithm aims to minimize an objective function known as the squared error function, calculated by the following formula:

number

[0118] Algorithmic Steps for K-Means Clustering The K-means clustering algorithm is as follows:

[0119] In K-means clustering, "c" cluster centers are randomly selected, the distance between each data point and the cluster center is calculated, the data point is assigned to the closest cluster center, and new cluster centers are recalculated using the following formula:

number

[0120] The distance between each data point and the newly obtained cluster center is measured, and if a data point is reassigned, the process continues until no data points are reassigned.

[0121] AI application processing flow In some embodiments, Digital Imaging and Communications in Medicine (DICOM) images are submitted through LION and transferred to the V2 platform via Hypertext Transfer Protocol Secure (HTTPS) by the DICOM ToolKit library (Offis.de DCMTK library). The DICOM images are temporarily stored on the V2 platform. In some embodiments, a DICOM record is created with limited information and the status is set to zero. In various embodiments, once the DICOM images are available in temporary storage, the V2 PHP / Laravel application begins processing the DICOM images via a Cron job.

[0122] In some embodiments, Cron job (1) monitors V2 for new DICOM images, retrieves the DICOM images from temporary storage, extracts tags, extracts frames (single subimage or multiple subimages), saves the images and tags to a data store, and sets a processing status to a database. In some embodiments, Cron job (1) converts and compresses the DICOM images to lossless JPG format using the Offis.de library DCMTK and sets a processing status to 1. In some embodiments, Cron job (1) runs automatically every few minutes, such as every 5 minutes, 4 minutes, 3 minutes, 2 minutes, or 1 minute. In some embodiments, Cron job (1) saves the DICOM image metadata to a table named DICOM in Microsoft SQL Server, extracts the images / frames, and stores the images / frames in a directory for the image manager. In various embodiments, records containing additional information about the images and the case IDs associated with the images are created during processing. The record contains additional data such as the subject's physical examination findings, a review of the subject's multiple visits, the series of images acquired during each exam, and the hierarchy of images within the case.

[0123] In various embodiments, the DICOM images and metadata are processed by the VetImages application, which is written in the PHP Laravel Framework. V2 makes a REST service request to VetImages to process each image asynchronously. In some embodiments, VetImages responds immediately to V2 to confirm that the request was received and that the process for cropping and evaluating the image continues in the background. Because images are processed in parallel, the overall process runs quickly.

[0124] In various embodiments, VetImages passes or forwards images to a module called VetConsole, which is written in Python and uses the computer vision technology OpenCV to preprocess the images. VetConsole identifies body parts in the images, such as the chest, abdomen, and pelvis, as a fallback in case the AI ​​cropping server is unable to classify the body part in the image. VetImages rotates and flips the image until the correct orientation is achieved. In some embodiments, VetImages uses an image match server to examine different angles and projections of the image. In various embodiments, the image match server is written in Python and Elastic Search to identify image matches. In some embodiments, the image database for the image match server is carefully selected to return results only if the image is found to be in the correct orientation and projection.

[0125] In various embodiments, once the image orientation is determined, VetImages sends a REST API request to a Keras / TensorFlow server to classify and identify body part regions of interest within the image. The VetImages REST API request is validated with an ImageMatch server to ensure that the returned image regions are classified as body parts such as chest, abdomen, pelvis, knee joint, etc. In some embodiments, if the cropping evaluation result is invalid, the VetConsole cropping result is validated and utilized.

[0126] In various embodiments, the AI ​​evaluation process that generates the AI ​​report begins when VetImages identifies the image as containing a classified and verified body part. In an alternative embodiment, if VetImages identifies the image as not containing a classified and verified body part, the image cropping process ends without results and without generating a report.

[0127] In various embodiments, VetImages sends a Keras / TensorFlow REST service call with the classified cropped images to an AI disease assessment model hosted on a TensorFlow application server written in python / Django, and stores the results of the AI ​​assessment model for final assessment and report generation.

[0128] VetImages also routes the chest cropped images to a TensorFlow server to determine whether the images are well-aligned relative to the parameters set by the user. VetImages sends the results of its AI assessment model to the V2 platform, which notifies the clinic of the results of the alignment assessment for each image.

[0129] In some embodiments, VetImages waits until all images for a case have been cropped and evaluated by the TensorFlow server while images are processed in parallel. In some embodiments, once all images for a case have been evaluated, VetImages processes all results for the case using rules defined by experts to identify report content in a more human-readable manner. In an alternative embodiment, VetImages uses a clustering model created by the user to identify the content of the AI ​​report. In some embodiments, the AI ​​report is assembled using previous radiologist reports that are used to build cluster models in VetImages. In some embodiments, clustering is used to classify cases / images using prediction results from other diagnostic models using scikit-learn.

[0130] In some embodiments, when specialist rules or cluster models are used to identify the content of an AI report, VetImages checks the species of the case, and in some embodiments, the report is generated and sent to a V2 clinic only if the species of the case is identified by VetImages as Canidae.

[0131] In some embodiments, VetImages sends a request to the V2 platform to notify the clinic that a new AI report has been sent to the clinic. In some embodiments, the V2 platform verifies the license of the clinic administrator user or of the user. In various embodiments, V2 attaches a copy of the report within or to the case document so that the report is accessible from the V1 platform if the clinic has a valid license. In some embodiments, V2 sends an email notification to the clinic email containing one or more links so that the email recipient can conveniently and immediately open the generated report.

[0132] Additive AI Model The current practice for AI model retraining is to improve current model performance by adding additional data, retraining, testing, and then replacing the current model with the updated AI model (Figure 24). Under the current practice, it is best practice to leave the old model in place for a specific window or period, or until the model serves a specific number of requests. The new model is provided with the same data, and the results are analyzed and compared with the old model. Thus, a better-performing model is identified. If the new model's performance is satisfactory, it is deployed. The current practice is a typical example of A / B testing, which ensures that the model is validated against upstream data. Another best practice is to automate the model deployment procedure after retraining. Machine learning models are deployed to production environments using Kubernetes (K8s), an open-source system for automating the deployment, scaling, and management of containerized applications.

[0133] In current approaches, it is assumed that the performance of a newly trained classifier is improved by reducing unwanted "noise" or false positive data. The assumed labeling of data that is detected as unwanted "noise" by the current classifier, and the removal of this "noise" by the retrained classifier, arises from the incorrect assumption that "noise" is not valuable information in overall system performance.

[0134] In fact, in contrast to the above erroneous assumption, the "noise" detected by the current classifier is useful for distinguishing between data points with similar but not identical characteristics. The AI ​​classifier results from the current and new classifiers in a combined analysis can better recognize and classify data compared to either AI classifier alone.

[0135] The "noise" detected by current models is useful for distinguishing between data points with similar but not identical characteristics. Therefore, replacing current AI models may result in the loss of valuable data. The concept of learning in the human brain is based on knowledge foundations and iterative learning steps. Along the path of iterative learning, prior knowledge is not lost or forgotten; rather, it is built upon and adjusted as needed, adding to one's overall bank of knowledge and capabilities. Thus, the concept of building on prior knowledge applies to daisy-chain iteration or associative model deployment.

[0136] Embodiments of the methods described herein enable continuous training through iterative or related model deployment or daisy-chain iterative or related models rather than the current standard of model retraining and replacement (FIG. 25). These methods improve AI reporting performance, internal AI audits by systems, and thereby advance toward, if not create, practical artificial general intelligence (AGI) systems.

[0137] For example, an AI classifier built to detect people with blonde hair may be extremely broad and may detect any person with any portion of blonde hair. A retrained AI classifier may be more specific and may only detect people with 80% or more blonde hair. Under current industry best practices, the AI ​​classifier would be removed from service and replaced with the retrained classifier.

[0138] Scenario 3 in Table 1 herein demonstrates that erroneous results can be easily detected when results from both AI classifiers are evaluated against a single, newly trained AI classifier result. In Scenario 3, the old classifier has higher sensitivity but lower specificity compared to the new classifier. Thus, instances that the old classifier evaluates as positive for blonde hair are also evaluated as positive by the new classifier. However, a negative result from the old or sensitive classifier and a positive result for the new or specific classifier are erroneous results that are immediately detected and revealed in a multi-classifier system. Therefore, the methods provided herein include rules that can be implemented within a system to keep different versions of the same classifier running so that the system can more quickly detect outlier results.

[0139] [Table 1]

[0140] The method described herein demonstrates implementing the addition of derivative models to a production system. In an additive model approach, users understand that the first model in production has some limitations and retrain the model to improve deficiencies. As subsequent AI models are built, the original or first iterative model is maintained, and results from the first and subsequent AI models are included in an AI cluster analysis, as described in U.S. Patent Application Publication No. 2021 / 0202092 A1, which is incorporated herein by reference in its entirety. AI data results from the old and new models are added to a single row of results, and an AI fingerprint is created for the item being evaluated. Multiple AI fingerprints are then clustered together to group similar results together. This approach of adding related or derivative AI classifiers is defined as daisy-chaining related or iterative, or unrelated or non-iterative, AI classifiers with accompanying words, phrases, sentences, and paragraphs, and non-AI-derived data (Figures 26 and 27).

[0141] Figure 28 shows a visual example of both derived (same letter and subscript number) and non-derived (same letter but different subscript) data. A database holds all of the training data with relationships. The system allows data input (in this case, I, T, and OI), single or clustered, AI or non-AI data, derived or non-derived data to obtain example results from the database. In Figure 28, the approach is subdivided into components A, B, and C, shown by the enclosed boxes.

[0142] The components in Box A show a visual example of how clustered data is grouped and related to train the system (i.e., add information to the database) with derived or non-derived results (training data), as well as an output step where clustered derived groupings of images (I) or text (T) are input to the system and corresponding results are returned from the database as best exemplar results. In this case, derived data is shown as input in images (I) and text (T).

[0143] The components in Box B show a visual example showing the clustered inputs, derived data (input data), and corresponding results (outputs pulled from the database) in a stacked visual arrangement.

[0144] The components in box C show a visual example showing the clustered inputs (inputs) of derived data and the corresponding results (outputs) in a horizontal visual arrangement. Having only derived classifiers is not a prerequisite for clustering inputs or outputs; in this example, OI in the output is indicated by the subscript OI, which indicates non-derived data. A and OI BThe different sized boxes indicate different weightings of the results. Additionally, the data represented by I, T, and OI should be thought of as building blocks that when grouped together form the instructions for constructing an output. That output could be a pixel, a collection of pixels, an image or a collection of images grouped together, a color or a particular arrangement of colors, a word or several words, a partial sentence or a complete sentence, several sentences that together make a partial paragraph, a complete paragraph or several paragraphs, or an entire template, an article, essay, or other text-based output, a type of measurement, a number, a waveform, a formula, a recipe, an atomic structure.

[0145] Like DNA primers, both AI and non-AI data, derived and non-derived data linked together into a unique fingerprint through clustering, are groupings of results that, when applied against a complementary database and specific user-defined presets, serve as instructions for a specific output. The novel concept here is to utilize AI- and non-AI-derived input data in a structured, linear format to serve as one or more "primer" instructions for generating inferences from a vast database of previously stored "primer" results and for a computerized system built around the data fingerprint to act as a transcription mechanism that creates the output. This fingerprinting or clustering "primer" approach allows for infinite possibilities for output results, thereby creating an extremely robust system (Figure 28).

[0146] By daisy-chaining related or iterative classifiers into the AI ​​evaluation, the results are subjected to additional sanity checks (Figures 29A and 29B). The methods described herein demonstrate implementing the addition of derivative models to a production system. In an additive model approach, the user understands that the first model in production has some limitations and retrains the model to improve deficiencies. As subsequent AI models are built, the original or first iterative model is maintained, and results from the first and subsequent AI models are included in an AI cluster analysis, as described in U.S. Patent Application Publication No. 2021 / 0202092 A1, which is incorporated herein by reference in its entirety. The AI ​​data results from the old and new models are added to a single row of results, and an AI fingerprint is created for the item being evaluated. Multiple AI fingerprints are then clustered together to group similar results together. This approach of adding related or derivative AI classifiers is defined as daisy-chaining related or iterative AI classifiers.

[0147] By daisy-chaining related or iterative classifiers into the AI ​​evaluation, the results are subjected to additional sanity checks. The sanity checks allow the original classifier to cross-reference results from subsequent classifiers. Furthermore, the sanity checks also identify any newly introduced AI evaluations with more specific classifiers if the original or broader classifier has a negative result for a case and the specific or newly trained classifier has a positive result.

[0148] In the examples of Figures 29A and 29B, a general pulmonary classifier (GLC) is used with a derived classifier (GLCn), or in this specific example, GLC2, as a confirmation of each other. GLC2 is a more specific bronchial pattern classifier. The results of GLC are on the left, and the results of GLC2 are on the right. The results of the general and specific classifiers are used as an internal check on each other for findings. In these examples, the classifier results are expected to track each other, since example 8a has no abnormal findings and example 8b has an abnormal finding picked up by both GLC and its derived classifier, GLC2. Depending on the specific imaging findings and classifiers used, acceptable combinations of GLC and GLC2 are both negative (<0.5) for the finding, both positive (>0.5) for the finding, or GLC >0.5 and GLC2 <0.5. Any results with GLC <0.5 and GLC2 >0.5 are considered abnormal and a system error.

[0149] Methods for machine learning, clustering, and programming are fully explained in the following references: Shaw, Zed. Learn Python the Hard Way: A Very Simple Introduction to the Terrifyingly Beautiful World of Computers and Code. Addison-Wesley, 2017; Ramalho, Luciano. Fluent Python. O'Reilly, 2016; Atienza, Rowel. Advanced Deep Learning with TensorFlow 2 and Keras: Apply DL, GANs, VAEs, Deep RL, Unsupervised Learning, Object Detection and Segmentation, and More. Packt, 2020; Vincent, William S. Django for Professionals: Production Websites with Python & Django. Still River Press, 2020; Bradski, Gary R., and Adrian Kaehler. Learning OpenCV: O'Reilly, 2011; Battiti, Roberto, and Mauro Brunato. The LION Way: Machine Learning plus Intelligent Optimization, Version 2.0, April 2015. LIONlab Trento University, 2015; Pianykh, Oleg S. Digital Imaging and Communications in Medicine (DICOM) a Practical Introduction and Survival Guide. Springer Berlin Heidelberg, 2012; Busuioc, Alexandru. The PHP Workshop: a New, Interactive Approach to Learning PHP.Packt Publishing, Limited, 2019; Stauffer, Matt. Laravel - Up and Running: a Framework for Building Modern PHP Apps. O'Reilly Media, Incorporated, 2019; Kassambara, Alboukadel. Practical Guide to Cluster Analysis in R: Unsupervised Machine Learning. STHDA, 2017; and Wu, Junjie. Advances in K-Means Clustering: A Data Mining Thinking. Springer, 2012. Each of these references is incorporated herein by reference in its entirety.

[0150] It will be understood that any feature described with respect to any one of the embodiments may be used alone or in combination with other features described, and may also be used in combination with one or more features of the other of the embodiments, or in any combination of the other of the embodiments. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the invention as defined in the appended claims.

[0151] The present invention has been fully described and is further illustrated by the following claims. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific methods described herein. Such equivalents are within the scope of the present invention and the claims. The contents of all references cited in this application, including issued patents and published patent applications, are hereby incorporated by reference in their entirety.

Claims

1. 1. A method for obtaining additive AI results from a digital file, comprising: processing the digital file with at least one first artificial intelligence (AI) classifier and at least one second AI classifier, thereby obtaining a first assessment result and at least one second assessment result, respectively; directing the first evaluation result and the at least one second evaluation result to at least one synthesis processor; Creating clusters of AI classifier and non-AI classifier results in a database; and generating an output by utilizing at least one of the first evaluation result and the second evaluation result as instructions and comparing them with at least one dataset cluster in the database, thereby obtaining a deductive additive AI output result. method.

2. The method of claim 1 , further comprising measuring a distance from the additive AI result to an example result from the dataset cluster to obtain an additive AI cluster identification.

3. The method of claim 1 , wherein the dataset clusters further comprise matched written templates.

4. The method of claim 3 , further comprising assembling the additive AI cluster identifications and the matched written templates to obtain a report.

5. The method of claim 4 , further comprising displaying the report to a user.

6. The method of claim 1 , wherein the at least one second AI classifier is a derivative of the first AI classifier.

7. 7. The method of claim 6, wherein the second AI classifier is trained using at least a portion of the data used to train the first AI classifier.

8. The method of claim 6 , wherein the second AI classifier is related to the first AI classifier.

9. The method of claim 1 , wherein the first AI classifier is a generic classifier.

10. The method of claim 1 , wherein the second AI classifier is a specific classifier.

11. The method of claim 1 , wherein the first AI classifier is a specific classifier.

12. The method of claim 1 , wherein the second AI classifier is a generic classifier.

13. 10. The method of claim 1, further comprising repeating the step of training and comparing against a series of daisy-chained related or derivative AI classifiers.

14. 10. The method of claim 1, further comprising comparing the first evaluation results with the second evaluation results to compare AI results and test expected performance.

15. The method of claim 1 , further comprising adding the first evaluation result and the second evaluation result to a results database.

16. The method of claim 1 , further comprising creating database cluster entries using one or more derivative classifiers of the at least one first AI classifier and the first evaluation classifier.

17. 17. The method of claim 16, further comprising evaluating additional data inputs using the at least one first AI evaluation classifier and the one or more derivative classifiers of the first evaluation classifier, comparing them to the database cluster entries, and returning example results from the database.

18. 10. The method of claim 1, further comprising a system or user input that accepts data input in at least one form selected from a pixel, a collection of pixels, an image, a collection of images grouped together, a color, an arrangement of colors, a word, several words, a partial sentence, a complete sentence, multiple sentences that together create a partial paragraph, a complete paragraph, multiple paragraphs, a complete template, an article, an essay or other text-based output, a measurement, a number, a waveform, a formula, a recipe, and mixed data for analysis and storage in the database or for analysis and generation of an output report.

19. The method of claim 1 , further comprising obtaining the digital file before processing.

20. 10. The method of claim 1, further comprising converting an analog file to said digital file before processing.

21. The method of claim 1 , further comprising classifying the digital files by performing at least one of labeling, cropping, editing, and orienting the digital files before processing.

22. The method of claim 1 , comprising clustering both AI-derived and non-AI-derived data.

23. The method of claim 1 , further comprising applying standard mathematical formulas, rearrangements, or weightings to the AI ​​results.

24. The method of claim 1 , further comprising obtaining and aggregating evaluation results in the database prior to utilization.

25. 10. A system programmed to obtain additive AI results by any of the methods of claim 1, comprising: at least one first AI processor; at least one derivative AI processor derived from the first AI processor; an output device; system.

26. 26. The system of claim 25, further comprising at least one database library.

27. 26. The system of claim 25, further comprising a user interface.

Images