Image Scoring Using Error Prediction

Abstract

A method (600) for scoring an image using error prediction includes receiving a frame (20) of image data representing the intensity of a state. The method further includes processing the frame of image data with a neural network (110). The method further includes predicting, by the neural network, an intensity value (120) of the state based on the frame of image data. The method further includes predicting, by the neural network, an error value (130) of the predicted intensity value. The error value represents a difference between the predicted intensity value and a ground truth intensity value (122) for the frame of image data.

Description

[Technical Field]

[0001] This disclosure relates to scoring some or all of an image using error prediction. [Background technology]

[0002] The advent of machine learning (e.g., neural networks) has enabled machines to evaluate frames of image data instead of, or in parallel with, manual evaluation. Typically, these algorithms receive input frames of image data, process the frames, and output a prediction. In some examples, the prediction is a score representing an aspect or feature of the frame. In other examples, the prediction is a segmentation of a region of interest in the frame of image data. In either case, machine learning allows for rapid processing of large amounts of image data. These techniques are applicable to a wide range of technologies. In one example, algorithms are used to analyze image data acquired during medical procedures (e.g., endoscopy, x-rays, etc.). Specifically, machine learning can be used to predict the intensity of a condition represented in a frame of image data acquired during a medical procedure (e.g., a medical procedure of an internal organ such as the digestive tract). Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present invention provides a method for scoring images using error prediction. The method includes receiving, in data processing hardware, frames of image data representing intensities of states. The method further includes processing, by the data processing hardware, the frames of image data using a neural network, and predicting, by the neural network, intensity values ​​of the states based on the frames of image data. The method further includes predicting, by the neural network, an error value of the predicted intensity value. The error value represents a difference between the predicted intensity value and a ground truth intensity value for the frame of image data.

[0004] Implementations of the present disclosure may include one or more of any of the following features: In some implementations, the method further includes receiving, by data processing hardware, video data including a series of frames of image data, each frame representing the intensity of the condition; and determining, by the data processing hardware, an intensity of the condition based on a predicted intensity value and a prediction error value for each frame of the series of frames of image data.

[0005] In some examples, determining the intensity of the state based on the predicted intensity value and the prediction error value for each frame of the series of frames includes fitting a curve to at least a portion of the predicted intensity values. Optionally, the method further includes determining, by data processing hardware, a frame informativeness of each frame in the series of frames of image data based on the frame's corresponding prediction error value, where frame informativeness represents the difficulty of determining the intensity value of the corresponding frame. The method further includes selecting, by the data processing hardware, a subset of frames of the series of frames of image data based on the frame informativeness, and providing, by the data processing hardware, the selected subset for manual evaluation.

[0006] In some implementations, selecting the subset of frames includes selecting frames with the highest frame informativeness. The neural network may include a recurrent convolutional neural network (CNN). Predicting the intensity value of the state based on the frames of image data may include predicting an orientation intensity value of the state for each of a plurality of different orientations of the frames of image data, and determining the intensity value of the state based on each of the orientation intensity values.

[0007] In some examples, determining an intensity value for the state based on each of the orientation intensity values ​​includes determining an inverse variance weighted average of the orientation intensity values. Optionally, the plurality of different orientations includes eight different orientations. In some implementations, the neural network is initially trained using a first loss function. The first loss function represents a difference between predicted intensity values ​​and ground truth intensity values ​​for the frame of image data. After the neural network is initially trained using the first loss function, it is trained using a second loss function that represents a difference between predicted intensity values ​​and ground truth intensity values ​​for the frame of image data and a difference between predicted error values ​​and ground truth error values ​​for the frame of image data.

[0008] In some embodiments, the condition comprises a disease. The frames of image data represent the interior surfaces of the patient's digestive tract. The digestive tract may include the esophagus, stomach, small intestine, and large intestine.

[0009] Another aspect of the present disclosure provides a method for scoring images using error prediction. The method includes receiving, in data processing hardware, a frame of image data representing a region of interest. The method further includes processing, by the data processing hardware, the frame of image data using a neural network, and segmenting, by the neural network, the region of interest based on the frame of image data. The method further includes predicting, by the neural network, an error value for the segmented region of interest. The error value represents a difference between the segmented region of interest in the frame of image data and a ground truth region of interest.

[0010] Another aspect of the present disclosure provides a system for providing image scoring using error prediction. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a frame of image data representing an intensity of a state. The operations further include processing the frame of image data using a neural network and predicting, by the neural network, an intensity value of the state based on the frame of image data. The operations further include predicting, by the neural network, an error value of the predicted intensity value. The error value represents a difference between the predicted intensity value and a ground truth intensity value for the frame of image data.

[0011] This aspect may include one or more of any of the following features: In some implementations, the operations further include receiving video data including a series of frames of image data, each frame representing an intensity of a condition, and determining an intensity of the condition based on a predicted intensity value and a prediction error value for each frame of the series of frames of image data.

[0012] In some examples, determining the intensity of the state based on the predicted intensity value and the prediction error value for each frame of the series of frames includes fitting a curve to at least a portion of the predicted intensity values. Optionally, the operations further include determining a frame informativeness for each frame in the series of frames of image data based on the frame's corresponding prediction error value. The frame informativeness represents the difficulty of determining the intensity value for the corresponding frame. The operations further include selecting a subset of frames of the series of frames of image data based on the frame informativeness and providing the selected subset for manual evaluation.

[0013] In some implementations, selecting the subset of frames includes selecting frames with the highest frame informativeness. The neural network may include a recurrent convolutional neural network (CNN). Predicting the intensity value of the state based on the frames of image data may include predicting an orientation intensity value of the state for each of a plurality of different orientations of the frames of image data, and determining the intensity value of the state based on each of the orientation intensity values.

[0014] In some examples, determining an intensity value for the state based on each of the orientation intensity values ​​includes determining an inverse variance weighted average of the orientation intensity values. Optionally, the plurality of different orientations includes eight different orientations. In some implementations, the neural network is initially trained using a first loss function. The first loss function represents a difference between predicted intensity values ​​and ground truth intensity values ​​for the frame of image data. After the neural network is initially trained using the first loss function, it is trained using a second loss function that represents a difference between predicted intensity values ​​and ground truth intensity values ​​for the frame of image data and a difference between predicted error values ​​and ground truth error values ​​for the frame of image data.

[0015] In some embodiments, the condition comprises a disease. The frames of image data represent the interior surfaces of the patient's digestive tract. The digestive tract may include the esophagus, stomach, small intestine, and large intestine.

[0016] Another aspect of the present disclosure provides another system for providing image scoring using error prediction. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a frame of image data representing a region of interest. The operations further include processing the frame of image data using a neural network and segmenting the region of interest based on the frame of image data by the neural network. The operations further include predicting, by the neural network, an error value for the segmented region of interest. The error value represents a difference between the segmented region of interest in the frame of image data and a ground truth region of interest.

[0017] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a schematic diagram of an exemplary system for scoring images using error prediction. [Figure 2] FIG. 2 is a schematic diagram of an intensity determiner of the exemplary system of FIG. 1. [Figure 3] FIG. 2 is a schematic diagram of an informative curator of the exemplary system of FIG. 1. [Figure 4] FIG. 2 is a schematic diagram of a frame orienter of the exemplary system of FIG. 1. [Figure 5] FIG. 2 is a schematic diagram of a loss rate function for the exemplary system of FIG. 1. [Figure 6] 1 is a flowchart of an example arrangement of operations of a method for performing image scoring user error prediction. [Figure 7] 10 is a flowchart of another exemplary arrangement of operations for a method of image scoring using error prediction. [Figure 8]FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0019] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0020] Analyzing frames of image data (e.g., to assess the intensity of a condition) has traditionally been a time-consuming task. However, advances in machine learning offer the opportunity to significantly reduce the effort required to analyze image data. Traditional machine learning algorithms are typically trained to predict values ​​(or segmentations of regions of interest) based on one or more frames of image data. A properly trained model can approach or even exceed the accuracy of a trained human rater.

[0021] In the medical field, large amounts of imaging data must be accurately analyzed. For example, a common medical procedure involves acquiring imaging data (e.g., video data) of organs and other structures within the body. This imaging data may be evaluated to determine the intensity of a condition. For example, a trained evaluator might evaluate imaging data of the digestive tract to determine the severity of diseases such as celiac disease or Crohn's disease. However, the wide range of possible symptoms and the varying quality of imaging data make it difficult for trained experts and machine learning algorithms to accurately and consistently assess a condition. However, accurate evaluation of imaging data is crucial for improving patient care.

[0022] Embodiments herein are directed to an image scoring system that includes an error prediction model that evaluates frames of image data and predicts at least one value based on the frames of image data and predicts an error in the predicted value. The prediction error represents the difference between the predicted value associated with the frame of image data and a ground truth value. That is, the error prediction model predicts an error in its own prediction. The system can use the predicted value and / or prediction error, for example, to determine the strength and certainty of a condition and / or to curate a set of frames of image data for further evaluation.

[0023] Referring to FIG. 1 , in some embodiments, an exemplary system 100 includes a computing device 10. The computing device 10 can correspond to any device, such as a desktop workstation, a laptop workstation, or a mobile device (i.e., a smartphone). The computing device 10 includes computing resources 12 (e.g., data processing hardware) and / or storage resources 14 (e.g., memory hardware). The computing device 10 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) with scalable / flexible computing and / or storage resources. The computing device 10 is configured to receive or acquire frames of image data 20. A frame may include any structure for encapsulating any type or amount of image data. The frames of image data 20, in some examples, represent the intensity of a condition. For example, the frames of image data 20 represent the inner surface of a patient's digestive tract (e.g., the esophagus, stomach, small intestine, or large intestine), and the condition is a disease. In other examples, the frames of image data 20 represent a region of interest. Computing device 10 may receive frames of image data 20 from another computing device via a peripheral device (eg, a camera or storage device) connected to computing device 10 or over a network, for example.

[0024] In some implementations, computing device 10 preprocesses frame of image data 20. For example, computing device 10 crops frame of image data 20 (e.g., to a 512x512 resolution), converts frame of image data 20 to grayscale (e.g., using the geometric mean of the RGB color channels), and / or encodes positional information within frame 20. In other examples, computing device 10 receives frame of image data 20 that has already been preprocessed by a remote entity.

[0025] Computing device 10 executes error prediction model 110, which in some implementations is a neural network having one or more layers. For example, error prediction model 110 is a recurrent convolutional neural network (CNN), although other types of neural networks can also be used. During inference, error prediction model 110 receives frame 20 of image data and, in some examples, predicts intensity values ​​120 of states represented in frame 20 of image data and predicts error values ​​130 of predicted intensity values ​​120. Prediction error values ​​130 represent the difference (i.e., the ideal result) between the predicted intensity values ​​and ground truth intensity values ​​122 ( FIG. 5 ) for frame 20 of image data.

[0026] The error value 130 may indicate the certainty or confidence of the error prediction model 110 in the predicted intensity value 120. For example, a small error value 130 (i.e., the error prediction model 110 predicts a small difference between the predicted intensity value 120 and the ground truth intensity value 122) may indicate a high confidence in the predicted intensity value 120. On the other hand, a larger error value 130 (i.e., the error prediction model 110 predicts a large difference between the predicted intensity value 120 and the ground truth intensity value 122) may indicate a low confidence in the predicted intensity value 120.

[0027] In the illustrated example, error prediction model 110 receives a frame 20 of image data indicating the intensity of a condition and predicts both an intensity value 120 and an error value 130 associated with the frame 20 of image data, although this example is not intended to limit the error prediction model 110. That is, error prediction model 110 may be trained to receive any type of data and make any prediction regarding the received data and predicted error value of the prediction. For example, error prediction model 110 may receive a frame of image data representing a region of interest. Error prediction model 110 may segment the region of interest (i.e., determine the location and boundaries of the region of interest) based on the frame of image data. Error prediction model 110 may also predict an error value for the segmented region of interest, representing the difference between the segmented region of interest and a ground truth region of interest. Again, the error value indicates the confidence or certainty of the predicted value, regardless of the details of the input data.

[0028] Continuing with reference to FIG. 1 , in some implementations, the error prediction model 110 is trained with training data 150 generated from a corpus of training frames of image data 152, where each training frame of image data 152 represents the intensity of a state. The corpus of training frames of image data 152 may be annotated (i.e., indicating the ground truth intensity values ​​associated with each frame of image data). During training, the error prediction model 110 receives the training data 150 and generates predicted intensity values ​​120 and error values ​​130. The predicted values ​​120, 130 and the actual ground truth values ​​122, 132 (from the annotated images 152) are fed back to the error prediction model 110 in the form of one or more loss functions ( FIG. 5 ) to modify one or more weights (e.g., backpropagation) associated with the error prediction model 110.

[0029] Referring now to FIG. 2 , in some implementations, the computing device 10 executes an intensity determiner 200 that includes an error prediction model 110. The intensity determiner 200 receives video data including a series of frames of image data 20a-n, each frame 20 representing the intensity of a state. The intensity determiner 200 provides the frames of image data 20 to the error prediction model 110 one at a time and records or stores the associated predicted intensity value 120 and error value 130 for each frame 20. The intensity determiner 200 determines an overall intensity of the state 212 based on the predicted intensity value 120 and / or the predicted error value 130 for each frame of the series of frames 20 of image data. For example, the intensity determiner 200 executes a curve fitter 210 that fits a curve to at least a portion of the predicted intensity values ​​120. The curve fitter 210 may use a weighted average of the frames 20. The weight of each predicted intensity value 120 may be based on the corresponding predicted error value 130. For example, a lower prediction error value 130, which may indicate a higher confidence in the prediction strength value 120, may be associated with a higher weight for the corresponding prediction strength value 120. Similarly, a higher prediction error value 130, which may indicate a lower confidence in the prediction strength value 120, may be associated with a lower weight for the corresponding prediction strength value 120.

[0030] Referring now to FIG. 3 , in some examples, a computing device executes an informativeness curator 300 that includes an error prediction model 110. The informativeness curator 300 includes an informativeness determiner 310 that receives prediction error values ​​130 for a series of frames 20 of image data (i.e., video data). The informativeness determiner 310 determines a respective frame informativeness 312 for each frame 20 of image data, which represents the difficulty of determining an intensity value for the corresponding frame 20 of the image data. The difficulty of determining the intensity value correlates with the error value 130 of the predicted intensity value 120. That is, a larger error value 130 may indicate less confidence in the predicted intensity value 120, which in turn may indicate greater difficulty in determining intensity values ​​from each frame 20 of the image data. For example, some frames 20 of the image data may be of poor quality making accurate assessment difficult, and the frame informativeness 312 can quantify this aspect.

[0031] The informativeness curator 300 may also implement a frame selector 320. The frame selector 320 selects a subset of frames of image data 20C from the set of frames of image data 20 based on the frame informativeness 312 of each frame 20 received from the informativeness determiner 310. For example, the frame selector 320 may select a number of frames 20C with the highest frame informativeness 312 (e.g., the 10 frames with the highest frame informativeness 312). In another example, the frame selector 320 selects all frames of image data 20C that meet a threshold informativeness value. For example, the frame selector 320 selects all frames 20C that have a frame informativeness 312 higher than the threshold.

[0032] In some examples, the frame selector 320 provides selected frames 20C for manual evaluation. This allows the informative curator 300 to curate a set of frames of image data 20C that are easiest (e.g., have the lowest error value 130) for determining the intensity of a condition. That is, the intensity of a condition is more evident in the curated set of frames 20C than in frames 20 of image data that were not selected by the informative curator 300. This allows the expert to quickly evaluate only the most relevant frames 20 of image data. In this way, the informative curator 300 can significantly reduce the number of frames 20 that the expert must evaluate to determine the intensity of a condition.

[0033] Referring now to FIG. 4 , in some examples, the computing device 10 executes a frame orienter 400 that receives a frame 20 of image data before the error prediction model 110. The frame orienter 400 provides the error prediction model 110 with each frame 20 of image data in a plurality of different orientations (e.g., rotated 90 degrees, 180 degrees, 270 degrees, etc.). For example, the frame orienter 400 provides the error prediction model 110 with eight different orientations of each frame 20 of image data. The error prediction model 110 predicts an orientation intensity value 120O and a corresponding orientation error value 130O for each of the plurality of different orientations. The computing device 10 can determine an overall intensity value for the frame 20 of image data based on each of the corresponding orientation intensity values ​​120O (and similarly, the orientation error values ​​130O). For example, the computing device 10 can determine the mean or inverse variance-weighted mean of the orientation intensity values ​​120O. The computing device 10 can perform other calculations based on the orientation intensity values ​​120O and the orientation error values ​​130O. For example, computing device 10 may determine the standard deviation of both the orientation intensity values ​​120O and the orientation error values ​​130O. The standard deviation may affect, among other things, the frame informativeness 312 of a frame 20 of image data. For example, a lower standard deviation may be associated with higher frame informativeness 312, and conversely, a higher standard deviation may be associated with lower frame informativeness 312.

[0034] Referring now to FIG. 5 , in some examples, the error prediction model 110 (i.e., a neural network) is trained with a combined loss function 530, which is a combination of an intensity loss function 510 and an error loss function 520. The intensity loss function 510 measures the loss or cost of a predicted intensity 120 relative to a corresponding ground truth intensity value 122. Generally, the closer the predicted intensity value 120 is to the ground truth intensity value 122, the smaller the intensity loss 512 determined by the intensity loss function 510. Similarly, the error loss function 520 determines the loss or cost of a predicted error value 130 relative to a ground truth error value 132. The ground truth error value 132 may be determined from the actual difference between the predicted intensity value 120 and the ground truth intensity value 122. The error loss function 520 determines the error loss 522 based on the difference between the predicted error value 130 and the ground truth error value 132.

[0035] The combination loss function 530 may receive both the intensity loss 512 and the error loss 522 and determine the combination loss 532. The combination loss 532 may be backpropagated to the error prediction model 110 to adjust the error prediction model 110. In some examples, the error prediction model 110 is first partially trained using only the intensity loss function 510 and the intensity loss 512. That is, the error prediction model 110 may be initially trained by backpropagating only the intensity loss 512. Following the initial training, the error prediction model 110 may be trained using the combination loss 532. In this manner, the error prediction model 110 is trained using a two-stage training process, primarily trained by the intensity loss 512 and then adjusted or tuned by the combination loss 532 to further improve accuracy. Optionally, in subsequent training, only a portion of the error prediction model 110 (e.g., only the top dense layer) may be trained first, and then the entire error prediction model 110 may be trained. In addition to other benefits of the error values ​​130, the inclusion of the error loss 522 (i.e., along with the combined loss function 530) can increase the accuracy of the predicted intensity values ​​120. That is, training the error prediction model 110 to predict its own error can directly lead to improved accuracy of the predicted intensity values ​​120 as measured against the ground truth intensity values ​​122.

[0036] Thus, the image scoring system provides an error prediction model 110 that not only predicts intensity values ​​but also predicts the error in the intensity values. The prediction error can be treated as a measure of confidence, helping the system ignore frames with low confidence and identify frames that are difficult for both machines and humans to evaluate. The techniques described here work well on regression outputs and do not require difficult and computationally expensive operations. The error prediction model can be trained end-to-end, simplifying training and integration within the system.

[0037] 6 is a flowchart of an exemplary configuration of operations for a method 600 of image scoring using error prediction. The method 600 includes, at step 602, receiving a frame of image data 20 representing the intensity of a state in data processing hardware 12. The method 600 includes, at step 604, processing the frame of image data 20 using a neural network 110 by the data processing hardware 12. At step 606, the method 600 includes predicting, by the neural network 110, an intensity value 120 of the state based on the frame of image data 20. At step 608, the method 600 includes predicting, by the neural network 110, an error value 130 of the predicted intensity value 120. The error value 130 represents the difference between the predicted intensity value 120 and a ground truth intensity value 122 for the frame of image data.

[0038] 7 is a flowchart of another exemplary configuration of operations for a method 700 of image scoring using error prediction. The method 700 includes, at step 702, receiving a frame 20 of image data representing a region of interest in the data processing hardware 12. The method 700 includes, at step 704, processing the frame 20 of image data by the data processing hardware 12 using a neural network 110. At step 706, the method 700 includes segmenting the region of interest based on the frame 20 of image data by the neural network 110. At step 708, the method 700 includes predicting, by the neural network 110, an error value 130 for the segmented region of interest. The error value 130 represents a difference between the segmented region of interest in the frame of image data and a ground truth region of interest.

[0039] 8 is a schematic diagram of an example computing device 800 that can be used to implement the systems and methods described herein. Computing device 800 is intended to represent digital computers in various forms, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are for illustrative purposes only and are not meant to limit the scope of the invention(s) described and / or claimed herein.

[0040] Computing device 800 includes a processor 810, memory 820, storage device 830, a high-speed interface / controller 840 connecting to memory 820, a high-speed expansion port 850, and a low-speed interface / controller 860 connecting to a low-speed bus 870 and storage device 830. Each of components 810, 820, 830, 840, 850, and 860 are interconnected using various buses and may be mounted on a common motherboard or otherwise as desired. Processor 810 processes instructions, including instructions stored in memory 820 or storage device 830, for execution within computing device 800 to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 880 coupled to high-speed interface 840. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as desired. Also, multiple computing devices 800 may be connected, each providing a portion of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0041] Memory 820 stores information non-transiently within computing device 800. Memory 820 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-transient memory 820 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by computing device 800. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0042] The storage device 830 can provide mass storage for the computing device 800. In some implementations, the storage device 830 is a computer-readable medium. In various different implementations, the storage device 830 may be a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 820, the storage device 830, or memory on the processor 810.

[0043] High-speed controller 840 manages bandwidth-intensive operations of computing device 800, while low-speed controller 860 manages less bandwidth-intensive operations. This assignment of duties is merely exemplary. In some implementations, high-speed controller 840 is coupled to memory 820, display 880 (e.g., via a graphics processor or accelerator), and to high-speed expansion port 850, which can accept various expansion cards (not shown). In some implementations, low-speed controller 860 is coupled to storage device 830 and low-speed expansion port 890. Low-speed expansion port 890, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled, for example, via a network adapter, to one or more input / output devices, such as a keyboard, pointing device, scanner, or a networking device such as a switch or router.

[0044] Computing device 800, as shown, may be implemented in many different forms, such as as a standard server 800a, or multiple times within a group of such servers 800a, as a laptop computer 800b, or as part of a rack server system 800c.

[0045] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive or transmit data and instructions from or to a storage system, at least one input device, and at least one output device.

[0046] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Examples of applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0047] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0048] The processes and logic flows described herein can be performed by one or more programmable processors, also known as data processing hardware, executing one or more computer programs that perform functions by manipulating input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to them. However, a computer need not include such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0049] To provide for user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to a user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be acoustic, verbal, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0050] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. 1. A method (600) comprising: receiving, in data processing hardware (12), a frame of image data (20) representing the intensity of a condition; processing the frames of image data (20) using a neural network (110) by the data processing hardware (12); predicting, by the neural network (110) as a first direct output of the neural network (110) during an inference operation, an intensity value (120) of the state based on the frame (20) of image data; predicting, by the neural network (110) as a second direct output of the neural network (110) during the inference operation, an error value (130) of the predicted intensity value (120), the prediction error value (130) representing a difference between the predicted intensity value (120) and a ground truth intensity value (122) for the frame (20) of image data; The method comprising:

2. receiving, at the data processing hardware (12), video data comprising a series of frames (20) of the image data, each frame (20) representing the intensity of the condition; determining, by the data processing hardware (12), the intensity of the state based on the predicted intensity value (120) and the predicted error value (130) for each frame (20) of the series of frames (20) of image data; The method (600) of claim 1, further comprising:

3. 3. The method of claim 2, wherein determining the intensity of the condition based on the predicted intensity values ​​and prediction error values ​​for each frame of the sequence of frames of image data comprises fitting a curve to at least a portion of the predicted intensity values.

4. determining, by the data processing hardware (12), a frame informativeness (312) for each frame (20) in the series of frames (20) of image data based on the corresponding prediction error value (130) for the frame (20), the frame informativeness (312) representing the difficulty of determining the intensity value (120) for the corresponding frame (20) of the image data; selecting, by the data processing hardware (12), a subset of frames (20) of the series of frames (20) of the image data based on the frame informativeness (312); providing, by the data processing hardware (12), a subset of the selected frames (20) for manual evaluation; The method (600) of claim 2, further comprising:

5. 5. The method (600) of claim 4, wherein selecting the subset of frames (20) comprises selecting the frames (20) of the image data having the highest frame informativeness (312).

6. 10. The method of claim 1, wherein the neural network comprises a recurrent convolutional neural network (CNN).

7. Predicting the intensity values ​​(120) of the states based on the frames (20) of image data includes: predicting an orientation intensity value (1200) of said state for each of a plurality of different orientations of said frame of image data (20); determining the intensity value (120) of the state based on each of the orientation intensity values ​​(120O); The method (600) of claim 1, further comprising:

8. 8. The method of claim 7, wherein determining the intensity value of the state based on each of the orientation intensity values ​​comprises determining an inverse variance weighted average of the orientation intensity values.

9. 8. The method (600) of claim 7, wherein the plurality of different orientations comprises eight different orientations.

10. 10. The method of claim 1, wherein the neural network is initially trained using a first loss function that represents the difference between the predicted intensity values ​​and the ground truth intensity values ​​for the frame of image data.

11. The neural network (110) is initially trained using the first loss function (510), and then: the difference between the predicted intensity value (120) and the ground truth intensity value (122) for the frame (20) of image data; a difference between the predicted error value (130) and a ground truth error value (132) for the frame of image data (20); 11. The method (600) of claim 10, wherein the training is performed using a second loss function (530) representing:

12. The method (600) of claim 1, wherein the condition comprises a disease.

13. 10. The method of claim 1, wherein the frames of image data represent the interior surfaces of a patient's digestive tract.

14. 14. The method (600) of claim 13, wherein the digestive tract includes the esophagus, stomach, small intestine, and large intestine.

15. The method of claim 1, wherein the frame of image data comprises a frame of medical image data.

16. 1. A method (700) comprising: receiving, in data processing hardware (12), a frame (20) of image data representing a region of interest; processing the frames of image data (20) using a neural network (110) by the data processing hardware (12); segmenting the region of interest based on the frame of image data by the neural network; predicting, by the neural network (110) as a direct output of the neural network (110) during an inference operation, an error value (130) for the segmented region of interest, the predicted error value (130) representing a difference between the segmented region of interest and a ground truth region of interest of the frame of image data; The method (700) comprising:

17. A system (100), comprising: data processing hardware (12); memory hardware (14) in communication with the data processing hardware (12), the memory hardware (14) being configured to, when executed on the data processing hardware (12), cause the data processing hardware (12) to: receiving a frame of image data (20) representing an intensity of a condition; processing said frames of image data (20) using a neural network (110); predicting, by the neural network (110) as a first direct output of the neural network (110) during an inference operation, an intensity value (120) of the state based on the frame (20) of image data; predicting, by the neural network (110) as a second direct output of the neural network (110) during the inference operation, an error value (130) of the predicted intensity value (120), the prediction error value (130) representing a difference between the predicted intensity value (120) and a ground truth intensity value (122) for the frame of image data; said memory hardware (14) storing instructions for performing operations including: The system (100) includes:

18. receiving video data including a series of frames (20) of the image data, each frame (20) representing the intensity of the condition; determining the intensity of the condition based on the predicted intensity value (120) and the predicted error value (130) for each frame (20) of the series of frames (20) of image data; The system (100) of claim 17, further comprising:

19. 20. The system (100) of claim 18, wherein determining the intensity of the condition based on the predicted intensity values ​​(120) and prediction error values ​​(130) for each frame (20) of the series of frames (20) of image data comprises fitting a curve to at least a portion of the predicted intensity values ​​(120).

20. determining a frame informativeness (312) for each frame (20) in the series of frames (20) of image data based on a corresponding prediction error value (130) for the frame (20), the frame informativeness (312) representing the difficulty of determining the intensity value (120) for the corresponding frame (20) of the image data; selecting a subset of frames (20) of the series of frames (20) of image data based on the frame informativeness (312); providing a subset (20) of the selected frames for manual evaluation; 20. The system (100) of claim 18, further comprising:

21. 21. The system (100) of claim 20, wherein selecting the subset of frames (20) comprises selecting the frames (20) of the image data having the highest frame informativeness (312).

22. 20. The system (100) of claim 17, wherein the neural network (110) comprises a recurrent convolutional neural network (CNN) (110).

23. Predicting the intensity values ​​(120) of the states based on the frames (20) of image data includes: predicting an orientation intensity value (1200) of said state for each of a plurality of different orientations of said frame of image data (20); determining the intensity value (120) of the state based on each of the orientation intensity values ​​(120O); The system (100) of claim 17, further comprising:

24. 24. The system (100) of claim 23, wherein determining the intensity value (120) of the state based on each of the orientation intensity values ​​(120O) comprises determining an inverse variance weighted average of the orientation intensity values ​​(120O).

25. 24. The system (100) of claim 23, wherein the plurality of different orientations comprises eight different orientations.

26. 20. The system of claim 17, wherein the neural network is initially trained using a first loss function that represents the difference between the predicted intensity values ​​and the ground truth intensity values ​​for the frame of image data.

27. The neural network (110) is initially trained using the first loss function (510), and then: the difference between the predicted intensity value (120) and the ground truth intensity value (122) for the frame (20) of image data; a difference between the predicted error value (130) and a ground truth error value (132) for the frame of image data (20); 27. The system (100) of claim 26, wherein the system (100) is trained using a second loss function (530) representing:

28. The system (100) of claim 17, wherein the condition comprises a disease.

29. 20. The system (100) of claim 17, wherein the frames of image data (20) represent an interior surface of a patient's digestive tract.

30. 30. The system (100) of claim 29, wherein the digestive tract includes the esophagus, stomach, small intestine, and large intestine.

31. 20. The system (100) of claim 17, wherein the frame of image data (20) comprises a frame of medical image data (20).

32. A system (100), comprising: data processing hardware (12); Memory hardware (14) in communication with data processing hardware (12), which, when executed on the data processing hardware (12), causes the data processing hardware (12) to: receiving a frame (20) of image data representing a region of interest; processing said frames of image data (20) using a neural network (110); segmenting the region of interest based on the frame of image data by the neural network; predicting, by the neural network (110) as a direct output of the neural network (110) during an inference operation, an error value (130) for the segmented region of interest, the predicted error value (130) representing a difference between the segmented region of interest and a ground truth region of interest of the frame of image data (20); said memory hardware (14) storing instructions for performing operations including: The system (100) includes:

Images