Systems and methods for training a generative adversarial network and using a trained generative adversarial network

Abstract

The present disclosure relates to systems and methods for training and using generative adversarial networks. In one embodiment, a system for training a generative adversarial network may include: at least one processor that may: provide a first plurality of images, the first plurality of images including representations of features of interest and indicators of locations of the features of interest; and train an object detection network using the first plurality of images and the indicators. In addition, the one or more processors may also: provide a second plurality of images including representations of the features of interest, and apply the trained object detection network to the second plurality of images to produce multiple detections of the features of interest. In addition, the one or more processors may also: provide manually set verification of true positives and false positives for multiple detections; train the generative adversarial network using the verification; and retrain the generative adversarial network using at least another set of images, further detections, and further manually set verifications.

Description

Technical Field

[0001] The present disclosure relates generally to the field of neural networks and to the use of such networks for image analysis and object detection. More specifically, but not by way of limitation, the present disclosure relates to systems and methods for training generative adversarial networks and computer-implemented systems and methods using the generative adversarial networks. The systems, methods, and trained neural networks disclosed herein can be used in a variety of applications and vision systems, such as medical image analysis and systems that benefit from accurate object detection capabilities. Background Art

[0002] In many object detection systems, objects are detected in images. Objects of interest can be people, places, or things. In applications such as medical image analysis and diagnosis, localization of objects is also important. However, computer-implemented systems that utilize image classifiers are generally unable to identify or provide the location of detected objects. Therefore, existing systems that use only image classifiers are not very useful.

[0003] In addition, training techniques for object detection will rely on manually annotated training sets. Such annotation is time-consuming when the trained detection network is a bounding box-based network (e.g., YOLO (You Only Look Once) architecture, single shot detector (SSD) architecture, etc.). Therefore, large data sets are difficult to annotate for training, which usually produces neural networks trained for smaller data sets, which reduces accuracy. For computer-implemented systems, existing medical imaging is typically built on a single detector network. Therefore, once a detection is performed, the network simply outputs the detection result, for example, to a doctor or other professional caregiver. However, such a detection result may be a false positive, such as a non-polyp in an endoscopy, etc. Such a system does not provide a separate network for distinguishing false positives from true positives.

[0004] In addition, neural network-based object detectors typically feed features identified by a neural network into a detector, which may include a second neural network. However, such networks are often inaccurate because feature detection is performed by a general-purpose network, while only the detector part is specialized.

[0005] Finally, many existing object detectors work with time delays. For example, medical images will be captured and stored before being analyzed. However, some medical procedures, such as endoscopy, are based on real-time diagnosis. Therefore, these systems are often difficult to apply in the required real-time manner. Summary of the invention

[0006] In view of the foregoing, embodiments of the present disclosure provide computer-implemented systems and methods for training generative adversarial networks and using them for applications such as medical image analysis. The systems and methods of the present disclosure provide benefits over existing systems and techniques, including improved object detection and location information.

[0007] According to some embodiments, a computer-implemented system is provided, the system including an object detector network that identifies features of interest (i.e., anomalies or objects of interest) and their locations, and an adversarial network that distinguishes true positives from false positives. In addition, embodiments of the present disclosure provide a two-loop technique for training object detector networks. The training process uses annotations based on inspection detection, so that manual annotations can occur faster and therefore have a larger data set. In addition, the process can be used to train a generative adversarial network to distinguish false positives from true positives.

[0008] In addition, a disclosed system is provided that combines an object detector network with a generative adversarial network. By combining these networks, false positives and true positives can be distinguished, thereby providing a more accurate output. By reducing false positives, doctors or other medical professionals will pay more attention to the output from the network due to the improved accuracy.

[0009] In addition, embodiments of the present disclosure include neural networks that do not use general feature recognition through one neural network combined with a specialized detector. Instead, a single seamless neural network is trained for the object detector portion, which results in better specialization and higher accuracy and efficiency.

[0010] Finally, embodiments of the present disclosure are configured to display real-time video (e.g., endoscopic video or other medical images) and object detection on a single display. Thus, embodiments of the present disclosure provide video bypass to minimize potential issues with errors and other potential shortcomings from object detectors. Furthermore, object detection can be displayed in a specially designed manner to better draw the attention of a physician or other professional caregiver.

[0011] In one embodiment, a system for training a generative adversarial network using images including representations of features of interest may include at least one memory configured to store instructions and at least one processor configured to execute the instructions to perform operations. The operations may include providing a first plurality of images including representations of the features of interest and indicators of locations of the features of interest in images of the first plurality of images; and training an object detection network using the first plurality of images and the indicators of the features of interest to detect the features of interest. The operations may also include providing a second plurality of images including representations of the features of interest, and applying the trained object detection network to the second plurality of images to produce a first plurality of detections of the features of interest. The second plurality of images may include a greater number of images than the number of images included in the first plurality of images. The operations may also include providing verification of a manual set of true positives and false positives for the first plurality of detections; training a generative adversarial network using verification of the first plurality of detections for true positives and false positives; and retraining the generative adversarial network using at least another set of images and detections of the features of interest and verification of further manual sets of true positives and false positives for further detections of the features of interest.

[0012] In some embodiments, the at least one processor may be further configured to retrain the generative adversarial network by providing verification of false negatives for missed detection of features of interest in two or more images.

[0013] In any embodiments, the object detection network can be a convolutional neural network.

[0014] In any embodiment, the number of images in the second plurality of images may be at least 100 times greater than the number of images included in the first plurality of images.

[0015] In any embodiment, the first plurality of images and the second plurality of images may include medical images. For example, the medical images may include images of gastrointestinal organs.

[0016] In any embodiment, at least one of the first plurality of images and the second plurality of images comprises an image from an endoscopic device. Additionally or alternatively, at least one of the first plurality of images and the second plurality of images comprises an image from an imaging device used during at least one of a gastroscopy, a colonoscopy, an enteroscopy, or an upper gastrointestinal endoscopy (e.g., an esophageal endoscopy).

[0017] In any embodiment, the feature of interest can be an anomaly. For example, the anomaly can include a change in human tissue from one type of cell to another type of cell. Additionally or alternatively, the anomaly can include a lack of human tissue from a location where human tissue is expected. Additionally or alternatively, the anomaly can include a formation on human tissue or a formation of human tissue.

[0018] In any embodiment, the abnormality may include a lesion. For example, the lesion may include a polypoid lesion or a non-polypoid lesion.

[0019] In one embodiment, a method for training a neural network system to detect anomalies in images of human organs may include: storing a plurality of videos including representations of anomalies in a database; selecting a first subset of the plurality of videos; and applying a perceptual branch of an object detection network to frames of the first subset of the plurality of videos to produce a first plurality of detections of anomalies. The method may also include: selecting a second subset of the plurality of videos; and training a generator network using the first plurality of detections and frames from the second subset of the plurality of videos to generate a plurality of artificial representations of the anomalies. The plurality of pseudo representations may be generated by residual learning. The method may also include: training an adversarial branch of a discriminator network to distinguish between pseudo representations of anomalies and true representations of anomalies; applying the adversarial branch of the discriminator network to a plurality of pseudo representations to produce a difference indicator between the pseudo representation of the anomaly included in the frames of the second subset of the plurality of videos and the true representation of the anomaly; applying the perceptual branch of the discriminator network to the pseudo representation to produce a second plurality of detections of the anomaly; and retraining the perceptual branch based on the difference indicator and the second plurality of detections. These steps may be performed by at least one processor.

[0020] In some embodiments, the abnormality may include a change in human tissue from one type of cell to another type of cell. Additionally or alternatively, the abnormality may include a lack of human tissue from a location where human tissue is expected. Additionally or alternatively, the abnormality may include a formation on human tissue or a formation of human tissue.

[0021] In any embodiment, the abnormality may include a lesion. For example, the lesion may include a polypoid lesion or a non-polypoid lesion.

[0022] In any embodiment, each of the pseudo-representations may provide a false representation of the anomaly that is highly similar to a true representation of the anomaly.

[0023] In any embodiments, the generator network may include a generative adversarial network.

[0024] In any embodiments, the discriminator network may include a convolutional neural network.

[0025] In one embodiment, a system for detecting features of interest in an image of a human organ may include: at least one memory storing instructions and at least one processor configured to execute the instructions to perform operations. The operations may include: selecting frames from a video of the human organ; applying a trained neural network system to the frames to produce at least one detection of the feature of interest; generating an indicator of the location of the at least one detection on one of the frames; re-encoding the frames into a video; and outputting the re-encoded video with the indicator. The neural network system may be trained according to any of the above embodiments.

[0026] Additional objects and advantages of the present disclosure will be set forth in part in the following detailed description, and in part will become apparent from the description, or may be learned through the practice of the present disclosure. The objects and advantages of the present disclosure will be realized and obtained by the elements and combinations particularly pointed out in the appended claims.

[0027] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which comprise a part of this specification, illustrate several embodiments and, together with the description, serve to explain the principles and features of the disclosed embodiments. In the drawings:

[0029] Figure 1 is a schematic representation of an exemplary computer-implemented system for overlaying object detection on a video feed according to an embodiment of the present disclosure.

[0030] Figure 2 is an exemplary two-stage training loop for an object detection network according to an embodiment of the present disclosure.

[0031] Figure 3 is a flowchart of an exemplary method for training an object detection network according to an embodiment of the present disclosure.

[0032] Figure 4 is a schematic representation of an exemplary object detector with a discriminator network and a generative network according to an embodiment of the present disclosure.

[0033] Figure 5 is a flow chart of an exemplary method for detecting features of interest using a discriminator network and a generator network according to an embodiment of the present disclosure.

[0034] Figure 6is a schematic representation of a computer-implemented system utilizing an object detector network according to an embodiment of the present disclosure.

[0035] Figure 7 is a flow chart of an exemplary method for overlaying object indicators on a video source using an object detector network according to an embodiment of the present disclosure.

[0036] Fig. 8A is an example of a display with an overlay for object detection in a video according to an embodiment of the present disclosure.

[0037] Figure 8B is another example of a display with an overlay for object detection in a video according to an embodiment of the present disclosure.

[0038] Figure 8C is an example of a display with an overlay for object detection in a video according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] The disclosed embodiments relate to computer-implemented systems and methods for training generative adversarial networks and using the generative adversarial networks. Advantageously, the exemplary embodiments can provide improved training networks and fast and efficient object detection. The disclosed embodiments can also provide improved object detection with reduced false positives for medical image analysis.

[0040] Embodiments of the present disclosure may be implemented and used in various applications and visual systems. For example, embodiments of the present disclosure may be implemented for medical image analysis systems and other types of systems that benefit from object detection, in which objects may be true positives or false positives. Although embodiments of the present disclosure are generally described herein with reference to medical image analysis and endoscopy, it should be understood that these embodiments may also be applied to other medical imaging processes, such as gastroscopy, colonoscopy, enteroscopy, and upper gastrointestinal endoscopy (e.g., esophageal endoscopy). In addition, embodiments of the present disclosure are not limited to other environments and visual systems, such as environments and visual systems for or including laser radar (LIDAR), monitoring, automatic guidance, and other imaging systems.

[0041] According to one aspect of the present disclosure, a computer-implemented system for training a generative adversarial network using an image including a representation of a feature of interest is provided. The system may include at least one memory configured to store instructions and at least one processor configured to execute the instructions (e.g., see Figure 1 and Figure 6). The at least one processor may provide a first plurality of images. For example, the at least one processor may extract the first plurality of images from one or more databases. Additionally or alternatively, the first plurality of images may include a plurality of frames extracted from one or more videos.

[0042] As used herein, the term "image" refers to any digital representation of a scene or field of view. The digital representation may be encoded in any suitable format, such as a joint photographic experts group (JPEG) format, a graphics interchange format (GIF), a bitmap format, a scalable vector graphics (SVG) format, an encapsulated PostScript (EPS) format, etc. Similarly, the term "video" refers to any digital representation of a scene or region of interest composed of multiple images in sequence. The digital representation may be encoded in any suitable format, such as a moving picture experts group (MPEG) format, a flash video format, an audio video interleave (AVI) format, etc. In some embodiments, the image sequence may be paired with audio.

[0043] The first plurality of images may include a representation of a feature of interest (i.e., an anomaly or an object of interest) and an indicator of the location of the feature of interest in the images of the first plurality of images. For example, the feature of interest may include an anomaly on human tissue or an anomaly of human tissue. In some embodiments, the feature of interest may include an object, such as a vehicle, a person, or other entity.

[0044] According to the present disclosure, "abnormality" may include: formation on or in human tissue, change of human tissue from one type of cell to another type of cell, and / or absence of human tissue from an expected human tissue location. For example, a tumor or other tissue growth may include an abnormality because there are more cells than expected. Similarly, a bruise or other change in cell type may include an abnormality because blood cells are present at a location other than the expected location (i.e., outside of capillaries). Similarly, a depression in human tissue may include an abnormality because cells are not present at the expected location, resulting in a depression.

[0045] In some embodiments, abnormality may include lesions. The lesions may include gastrointestinal mucosal lesions. Lesions may be classified histologically (e.g., according to the Vienna classification), morphologically (e.g., according to the Paris classification), and / or structurally (e.g., serrated or non-serrated). The Paris classification includes polypoid lesions and non-polypoid lesions. The polypoid lesions may include protruding lesions, pedunculated and protruding lesions, or pedunculated lesions. Non-polypoid lesions may include surface raised lesions, flat lesions, surface shallow depressed lesions, or depressed lesions.

[0046] For detected abnormalities, serrated lesions may include sessile serrated adenomas (SSA); traditional serrated adenomas (TSA); hyperplastic polyps (HP); fibroblastic polyps (FP); or mixed polyps (MP). According to the Vienna classification, abnormalities are divided into the following five categories: (category 1) negative for neoplasm / dysplasia; (category 2) indeterminate for neoplasm / dysplasia; (category 3) non-invasive low-grade malignancy (low-grade adenoma / dysplasia); (category 4) mucosal high-grade cancer, such as high-grade adenoma / dysplasia, non-invasive carcinoma (carcinoma in situ), or suspected invasive carcinoma; and (category 5) invasive tumor, intramucosal carcinoma, submucosal carcinoma, etc.

[0047] The indicator of the location of the abnormality or feature of interest may include a point (e.g., coordinates) or an area (e.g., a rectangle, square, ellipse, or any other regular or irregular shape). The indicator may include a manual annotation on the image or a manual annotation of the image. In some embodiments, the first plurality of images may include medical images, such as images of gastrointestinal organs or other organs or regions of human tissue. The image may be generated from a medical imaging device (e.g., a medical imaging device used during endoscopy, gastroscopy, colonoscopy, enteroscopy, or upper gastrointestinal endoscopy (e.g., esophageal endoscopy)). In this embodiment, if the feature of interest is a lesion or other abnormality, a doctor or other professional caregiver may annotate the image to place an indicator of the abnormality in the image.

[0048] One or more processors of the system may train an object detection network using the first plurality of images and the indicator of the feature of interest to detect the feature of interest. For example, the object detection network may include a neural network having one of a plurality of layers configured to accept as input an image and output an indicator of the location of the feature of interest. In some embodiments, the object detection network may include a convolutional network.

[0049] Training the object detection network may include adjusting the weights of one or more nodes of the network and / or adjusting the activation (or transfer) function of one or more nodes of the network. For example, the weights of the object detection network may be adjusted to minimize the loss function associated with the network. In some embodiments, the loss function may include a square loss function, a hinge loss function, a logistic loss function, a cross entropy loss function, any other appropriate loss function, or a combination of these loss functions. In some embodiments, the activation (or transfer) function of the object detection network may be modified to improve the fit between one or more models of one or more nodes and the input to one or more nodes. For example, one or more processors may increase or decrease the power of a polynomial function associated with one or more nodes, may change the associated function from one type to another (e.g., changing a polynomial to an exponential function, changing a logarithmic function to a polynomial, etc.), or perform any other adjustment to one or more models of one or more nodes.

[0050] The one or more processor systems may also provide a second plurality of images including representations of features of interest. For example, the one or more processors may extract the first plurality of images from one or more databases, whether the same one or more databases storing the first plurality of images or one or more different databases. Additionally or alternatively, the second plurality of images may include a plurality of frames extracted from one or more videos, whether the same videos or one or more different videos used to extract the first plurality of images.

[0051] In some embodiments, the second plurality of images may include medical images, such as images from an endoscopic device. In this embodiment, the features of interest may include lesions or other abnormalities.

[0052] In some embodiments, the second plurality of images may include more images than the number of images included in the first plurality of images. For example, the second plurality of images may include at least 100 times more images than the first plurality of images. In some embodiments, the second plurality of images may at least partially include the first plurality of images, or may be different images than the first plurality of images. In embodiments where the second plurality of images is extracted from one or more videos (from which at least a portion of the first plurality of images is at least partially extracted), the second plurality of images may include frames different from the first plurality of images from the same one or more videos.

[0053] One or more system processors may apply the trained object detection network to the second plurality of images to produce a first plurality of detections of features of interest. For example, in embodiments where the trained object detection network comprises a neural network, at least one processor may input the second plurality of images to the network and receive the detections. The detections may include indicators of locations of features of interest in the second plurality of images. If the second plurality of images do not include features of interest, the indicators may include a null indicator of no features of interest or other indicators.

[0054] One or more system processors may also provide verification of manually set true positives and false positives for the first plurality of detections. For example, the verification may be extracted from one or more databases or received as input. In embodiments where the feature of interest includes a lesion or other abnormality, the verification may be input by a physician or other professional caregiver. For example, one or more processors may output the detections for display to a physician or other professional caregiver, and receive verification in response to the displayed detections.

[0055] One or more system processors can use the verification of the first plurality of detection pairs of true positives and false positives to train a generative adversarial network. For example, a generative branch of the network can be trained to generate an artificial representation of the feature of interest. Thus, the generative branch can include a convolutional neural network.

[0056] Similar to the object detection network, training the generative branch can include adjusting the weights of one or more nodes of the network and / or adjusting the activation (or transfer) function of one or more nodes of the network. For example, as explained above, the weights of the generative branch can be adjusted to minimize the loss function associated with the network. Additionally or alternatively, the activation (or transfer) function of the generative branch can be modified to improve the fit between one or more models of one or more nodes and the input of one or more nodes.

[0057] In addition, the adversarial branch of the network can be trained based on manual verification to distinguish between false positives and true positives. For example, the adversarial branch can include a neural network that accepts an image and one or more corresponding detections as input and produces a verification as output. In some embodiments, one or more processors can further retrain the generative network by providing a verification of false negatives for missed detections of features of interest in two or more images. By providing the adversarial branch with a pseudo-representation from the generative branch as input and recursively utilizing the output from the adversarial branch, the adversarial branch and the generative branch can perform unsupervised learning.

[0058] Similar to generative branches, training adversarial branches can include adjusting weights of one or more nodes of the network and / or adjusting activation (or transfer) functions of one or more nodes of the network. For example, as explained above, the weights of the adversarial branches can be adjusted to minimize a loss function associated with the network. Additionally or alternatively, the activation (or transfer) function of the adversarial branch can be modified to improve the fit between one or more models of one or more nodes and the input of one or more nodes.

[0059] Thus, in embodiments where the features of interest include lesions or other abnormalities, the generative branch can be trained to generate representations of non-anomalies that appear similar to anomalies, and the adversarial branch can be trained to distinguish between pseudo non-anomalies and anomalies in the second plurality of images.

[0060] One or more system processors may retrain the generative adversarial network using at least another set of images and detection of features of interest and further manually set verification of true positives and false positives regarding further detection of features of interest. For example, one or more processors may extract another set of images from one or more databases, whether the same one or more databases storing the first plurality of images and / or the second plurality of images or one or more different databases. Additionally or alternatively, the another set of images may include multiple frames extracted from one or more videos, whether the same video or one or more different videos used to extract the first plurality of images and / or the second plurality of images. Similar to training, retraining the adversarial branch may include further adjustments to the weights of one or more nodes of the network and / or further adjustments to the activation (or transfer) functions of one or more nodes of the network.

[0061] According to another aspect of the present disclosure, a computer-implemented method for training a neural network system to detect anomalies in images of human organs is provided. The method may be performed by at least one processor (e.g., see Figure 6 The processor 607) is used to implement it.

[0062] According to an exemplary method, one or more processors may store a plurality of videos including abnormal representations in a database. For example, the video may include an endoscope video. The video may be encoded into one or more formats, such as a Moving Picture Experts Group (MPEG) format, a flash video format, an audio video interleave (AVI) format, etc.

[0063] The method may also include: selecting, using one or more processors, a first subset of the plurality of videos. For example, the one or more processors may randomly select the first subset. Alternatively, the one or more processors may select the first subset using one or more indexes of a database. For example, the one or more processors may select the first subset as a video indexed as including an abnormal representation.

[0064] The method may also include applying, using one or more processors, a perception branch of an object detection network to frames of a first subset of the plurality of videos to produce a first plurality of detections of anomalies. For example, the object detection network may include a neural network trained to accept an image as input and output a first plurality of detections. The first plurality of detections may include indicators of locations of anomalies in the frame, such as points or areas of detected anomalies. The lack of anomalies will result in a null indicator of no anomalies or other indicators. The perception branch may include a neural network (e.g., a convolutional neural network) configured to detect anomalies and output indicators of the locations of any detected anomalies.

[0065] The method may also include: selecting, using one or more processors, a second subset of the plurality of videos. In some embodiments, the second subset may at least partially include the first subset, or may be different videos from the first subset.

[0066] The method may also include training a generator network using the first plurality of detections and frames from a second subset of the plurality of videos to generate a plurality of pseudo representations of the anomalies. For example, the generator network may include a neural network configured to generate the pseudo representations. In some embodiments, the generator network may include a convolutional neural network. The plurality of pseudo representations may be generated by residual learning.

[0067] As explained above, training the generative network can include adjusting the weights of one or more nodes of the network and / or adjusting the activation (or transfer) function of one or more nodes of the network. For example, as described above, the weights of the generative network can be adjusted to minimize a loss function associated with the network. Additionally or alternatively, the activation (or transfer) function of the generative network can be modified to improve the fit between one or more models of one or more nodes and the input of one or more nodes.

[0068] The method may also include: training an adversarial branch of the discriminator network using one or more processors to distinguish between anomalous pseudo-representations and anomalous true representations. For example, the adversarial branch may include: a neural network that accepts as input a representation and outputs an indication of whether the input representation is a pseudo-representation or a true representation. In some embodiments, the neural network may include a convolutional neural network.

[0069] Similar to the generative branch, training the adversarial branch of the discriminator network can include: adjusting the weights of one or more nodes of the network and / or adjusting the activation (or transfer) function of one or more nodes of the network. For example, as explained above, the weights of the adversarial branch of the discriminator network can be adjusted to minimize the loss function associated with the network. Additionally or alternatively, the activation (or transfer) function of the adversarial branch of the discriminator network can be modified to improve the fit between one or more models of one or more nodes and the input of one or more nodes.

[0070] The method may also include: applying the adversarial branch of the discriminator network to multiple pseudo representations using one or more processors to generate a difference indicator between the pseudo representation of the anomaly and the true representation of the anomaly included in the frames of the second subset of the plurality of videos. For example, the pseudo representation may include a non-anomalous representation that looks similar to the anomaly. Thus, each pseudo representation may provide a pseudo representation of the anomaly that is highly similar to the true representation of the anomaly. The adversarial branch may learn to identify the difference between the non-anomalous (false representation), particularly the non-anomalous that is similar to the anomaly and the anomaly (true representation).

[0071] The method may also include: applying the perception branch of the discriminator network to the pseudo representation using one or more processors to produce a second plurality of detections of anomalies. Similar to the first plurality of detections, the second plurality of detections may include indicators of locations of anomalies in the pseudo representation, such as points or regions of detected anomalies. The lack of anomalies may result in a null indicator of no anomalies or other indicators.

[0072] The method may also include retraining the perception branch based on the difference indicator and the second plurality of detections. For example, retraining the perception branch may include adjusting weights of one or more nodes of the network and / or adjusting activation (or transfer) functions of one or more nodes of the network. For example, as explained above, the weights of the perception branch may be adjusted to minimize a loss function associated with the network. Additionally or alternatively, the activation (or transfer) function of the perception branch may be modified to improve the fit between the one or more models and the difference indicator of the one or more nodes and the second plurality of detections.

[0073] The exemplary method of training described above may produce a trained neural network system. The trained neural network system may form part of a system for detecting features of interest in images of human organs (e.g., the neural network system may be implemented as Figure 1 The system may include at least one memory configured to store instructions and at least one processor configured to execute the instructions. The at least one processor may select frames from a video of a human organ. For example, the video may include an endoscope video.

[0074] One or more system processors may apply the trained neural network system to the frame to produce at least one detection of a feature of interest. In some embodiments, the feature of interest may include an anomaly. The at least one detection may include an indicator of the location of the feature of interest. For example, the location may include a point or area at which the feature of interest is detected. The neural network system may be trained to detect anomalies as explained above.

[0075] In some embodiments, one or more system processors may also apply one or more additional classifiers and / or neural networks to the detected feature of interest. For example, if the feature of interest includes a lesion, at least one processor may classify the lesion into one or more types (e.g., cancer or non-cancerous, etc.). Additionally or alternatively, the neural network system may further output whether the detected feature of interest is a false positive or a true positive.

[0076] One or more system processors may generate an indicator of at least one detected location on a frame. For example, the location of the feature of interest may be extracted from the indicator and a graphical indicator of the location placed on the frame. In embodiments where the location includes a point, the graphical indicator may include a circle, a star, or any other shape placed on the point. In embodiments where the location includes an area, the graphical indicator may include a border around the area. In some embodiments, the shape or border may be animated; thus, a shape or border may be generated for multiple frames so that it tracks the location of the feature of interest across the frame and is animated when the frames are displayed sequentially. As further explained below, the graphical indicator may be paired with other indicators (e.g., sound and / or vibration indicators).

[0077] Any aspect of the indicator may depend on the classification of the feature of interest, for example, on the classification of the feature of interest as one or more types or as a false positive or true positive. Thus, the color, shape, pattern or other aspect of the graphical indicator may depend on the classification. In embodiments that also use sound and / or vibration indicators, the duration, frequency and / or amplitude of the sound and / or vibration may depend on the classification.

[0078] One or more system processors may re-encode the frames into a video. Thus, after generating the (graphical) indicator and superimposing it on one or more frames, the frames may be reassembled into a video. Thus, one or more processors of the system may output the re-encoded video with the indicator.

[0079] According to another aspect of the present disclosure, a computer-implemented system for processing real-time video is described (see, e.g., Figure 1 and Figure 6 ). The system may include an input port for receiving real-time video. For example, the input port may include a video graphics array (VGA) port, a high-definition multimedia interface (HDMI) port, a digital visual interface (DVI) port, a serial digital interface (SDI), etc. The real-time video may include medical video. For example, the system may receive real-time video from an endoscopic device.

[0080] The system may also include a first bus for transmitting the received real-time video. For example, the first bus may include a parallel connection or a serial connection, and may be wired into a multi-point topology or a daisy chain topology. The first bus may include a PCI Express (Peripheral Component Interconnect Express) bus, a universal serial bus (USB), an IEEE 1394 interface (FireWire), etc.

[0081] The system may include at least one processor configured to receive real-time video from a first bus, perform object detection on frames of the received real-time video, and superimpose a boundary in the frame indicating the location of at least one detected object. The one or more processors may perform object detection using a neural network system that is trained to produce at least one detection of an object. In some embodiments, the at least one object may include a lesion or other abnormality. Thus, the neural network system may be trained to detect abnormalities as explained above.

[0082] The one or more processors may superimpose a boundary as explained above. For example, the boundary may surround an area including an object, the area being received by at least one detection performed by the one or more processors.

[0083] The system may also include a second bus for receiving video with superimposed borders. For example, similar to the first bus, the second bus may include a parallel connection or a serial connection, and may be wired into a multi-point topology or a daisy chain topology. Thus, similar to the first bus, the second bus may include a PCI Express (Peripheral Component Interconnect Express) bus, a Universal Serial Bus (USB), an IEEE 1394 interface (Firewire), etc. The second bus may include a bus of the same type as the first bus, or may include a bus of a different type.

[0084] The system may also include an output port for outputting the video with the superimposed border from the second bus to an external display. The output port may include a VGA port, an HDMI port, a DVI port, an SDI port, etc. Thus, the output port may be a port of the same type as the input port, or may be a port of a different type.

[0085] The system may include a third bus, which is used for directly transmitting the received real-time video to an output port. The third bus can passively transport the real-time video from the input port to the output port, thereby starting even when the entire system is turned off. In certain embodiments, the third bus can be a default bus that is also started when the entire system is turned off. In this embodiment, when the entire system is started, the first bus and the second bus can be started, and the third bus can be not started accordingly. When the entire system is turned off, or once an error signal is received from one or more processors, the third bus can be restarted. For example, if the object detection implemented by the processor fails, then one or more processors can start the third bus, thereby allowing the continuous output of the real-time video stream, and can not be interrupted due to failure.

[0086] In some embodiments, the superimposed border can be modified across the frame. For example, the superimposed border can include a two-dimensional shape displayed around an area of ​​the image including the at least one detected object, the border being a first color. After a period of time, if the at least one detected object is a true positive, the one or more processors can modify the border to a second color, and if the at least one detected object is a false positive, the one or more processors can modify the border to a third color. Additionally or alternatively, the one or more processors can modify the border based on the classification of the detected object. For example, if the object includes a lesion or other abnormality, the modification can be based on whether the lesion or formation is a cancer or other abnormality.

[0087] In any of the above embodiments, the superimposed indicator can be paired with one or more additional indicators. For example, when at least one object is detected, one or more processors can send a command to one or more speakers to produce a sound. In an embodiment where the boundary is modified, one or more processors can send a command when the boundary is modified. In this embodiment, at least one of the duration, pitch, frequency, and amplitude of the sound can depend on whether the at least one object detected is a true positive or a false positive. Additionally or alternatively, at least one of the duration, pitch, frequency, and amplitude of the sound can depend on the classification of the detected object.

[0088] Additionally or alternatively, when at least one object is detected, the one or more processors may send a command to the at least one wearable device to vibrate. In an embodiment where the boundary is modified, the one or more processors may send the command when the boundary is modified. In this embodiment, at least one of the duration, frequency, and amplitude of the vibration may depend on whether the at least one detected object is a true positive or a false positive. Additionally or alternatively, at least one of the duration, pitch, frequency, and amplitude of the vibration may depend on the classification of the detected object.

[0089] According to another aspect of the present disclosure, a system for processing real-time video is described. Similar to the processing system described above, the system may include: an input port for receiving real-time video; at least one processor configured to receive the real-time video from the input port, perform object detection by applying a trained neural network to a frame of the received real-time video, and superimpose a border indicating the location of at least one detected object in the frame; and an output port for outputting the video with the superimposed border from the processor to an external display.

[0090] The system may also include an input device for receiving a sensitivity setting from a user. For example, the input device may include a key, one or more buttons, or any other device suitable for receiving a command to increase a setting and another command to decrease a setting.

[0091] One or more system processors may adjust at least one parameter of the trained neural network in response to the sensitivity setting. For example, one or more processors may adjust one or more weights of one or more nodes of the network based on the sensitivity setting to increase or decrease the number of detections generated by the network. Additionally or alternatively, one or more thresholds of an output layer of the network and / or one or more thresholds applied to detections received from the output layer of the network may be increased or decreased in response to the sensitivity setting. Thus, if the sensitivity setting is increased, one or more processors may decrease the one or more thresholds such that the number of detections generated by the network increases. Similarly, if the sensitivity setting is decreased, one or more processors may increase the one or more thresholds such that the number of detections generated by the network decreases.

[0092] Figure 1 is a schematic diagram of an exemplary system 100 including circuitry for overlaying object detection on a video feed consistent with an embodiment of the present disclosure. Figure 1 As shown in the example of , system 100 includes an operator 101 controlling an imaging device 103. In embodiments where the video source includes a medical video, operator 101 may include a physician or other professional caregiver. Imaging device 103 may include a medical imaging device, such as an X-ray machine, a computed tomography (CT) machine, a magnetic resonance imaging (MRI) machine, an endoscopy machine, or other medical imaging device that produces a video or one or more images of a human body or a portion of a human body. Operator 101 may control imaging device 103 by controlling the capture rate of device 103 and / or controlling the movement of device 103 (e.g., through or relative to a human body). In some embodiments, imaging device 103 may include a Pill-Cam TM Devices, or other forms of capsule endoscopic devices that replace external imaging devices, such as X-ray machines, or imaging devices inserted through a body cavity, such as endoscope devices.

[0093] like Figure 1 As further shown, the imaging device 103 can transmit the captured video or image to the overlay device 105. The overlay device 105 can include one or more processors to process the video, as described above. In addition, in some embodiments, in addition to controlling the imaging device 103, the operator 101 can also control the overlay device 105, for example, by controlling the sensitivity of an object detector (not shown) of the overlay device 105.

[0094] like Figure 1As shown, the overlay device 105 can augment the video received from the imaging device 103 and then transmit the augmented video to the display 107. In some embodiments, the augmentation can include the overlay described above. Figure 1 As further shown, the overlay device 105 may also be configured to forward the video from the imaging device 103 directly to the display 107. For example, the overlay device 105 may perform direct forwarding under predetermined conditions, such as if an object detector (not shown) included in the overlay device 105 fails. Additionally or alternatively, the overlay device 105 may perform direct forwarding if the operator 101 inputs a command to the overlay device 105. The command may be received via one or more buttons included on the overlay device 105 and / or via an input device such as a keyboard.

[0095] Figure 2 2 is a schematic representation of a two-stage training loop 200 for an object detection network consistent with an embodiment of the present disclosure. Loop 200 may be implemented by one or more processors. Figure 2 As shown, the first stage of loop 200 may use a database 201 of images including features of interest. In embodiments where the images include medical images, the features of interest may include abnormalities, such as lesions.

[0096] As explained above, the database 201 may store a single image and / or one or more videos, each video comprising a plurality of frames. During the first phase of the loop 200, the one or more processors may extract a subset 203 of images and / or frames from the database 201. The one or more processors may select the subset 203 randomly or at least in part using one or more patterns. For example, if the database 201 stores videos, the one or more processors may select no more than one, two, or a similar number of frames from each video included in the subset 203.

[0097] like Figure 2As shown, feature indicator 205 may include annotations to subset 203. For example, annotations may include points of features of interest or areas including features of interest. In some embodiments, an operator may view a video or image and manually input annotations to one or more processors via an input device (e.g., any combination of a keyboard, a mouse, a touch screen, and a display). The annotations may be stored as a data structure independent of the image, such as in a format such as JSON, XML, text, etc. For example, in an embodiment where the image is a medical image, the operator may be a doctor or other professional caregiver. Although described as being added to subset 203 after extraction, subset 203 may be annotated before being stored in database 201 or at other earlier times. In this embodiment, one or more processors may select subset 203 by selecting an image in database 201 having feature indicator 205.

[0098] The subset 203, together with the feature indicators 205, comprises a training set 207. The one or more processors may utilize the training set 207 to train a discriminator network 209. For example, the discriminator network 209 may comprise an object detector network as described above. As further explained above, training the discriminator network may comprise adjusting weights of one or more nodes of the network and / or adjusting activation (or transfer) functions of one or more nodes of the network. For example, the weights of the object detection network may be adjusted to minimize a loss function associated with the network. In another example, the activation (or transfer) function of the object detection network may be modified to improve the fit between one or more models of one or more nodes and the input of one or more nodes.

[0099] like Figure 2 As shown, during the second stage of loop 200, one or more processors may extract a subset 211 of images (and / or frames) from database 201. Subset 211 may at least partially include some or all of the images from subset 203, or may include a different subset. In embodiments where subset 203 includes multiple frames from one or more videos, subset 211 may include adjacent frames or other frames from one or more of the same videos. Subset 211 may include a greater number of images than subset 203, for example, subset 211 includes at least 100 times more images than subset 203.

[0100] One or more processors may apply discriminator network 209' (which represents discriminator network 209 after completing the first phase of training) to subset 211 to produce a plurality of feature indicators 213. For example, feature indicators 213 may include points or regions of features of interest detected by discriminator network 209'.

[0101] like Figure 2As further shown, verification 215 may include annotations of feature indicators 213. For example, the annotations may include an indicator of whether each feature indicator is a true positive or a false positive. Images that do not detect a feature of interest but include the feature of interest may be labeled as false negatives.

[0102] The subset 211, along with the feature indicators 213 and the validation 215, includes a training set 217. The one or more processors may use the training set 217 to train a generative adversarial network 219. For example, the generative adversarial network 219 may include a generative network and an adversarial network as described above. Training the generative adversarial network may include training the generative network to generate pseudo representations of features of interest or pseudo representations of false features of interest that look similar to real features of interest, and training the adversarial network to distinguish between pseudo representations and true representations, such as those included in the subset 211.

[0103] Despite Figure 2 Not shown, validation 213 may be further used to retrain discriminator network 209'. For example, the weights and / or activation (or transfer) functions of discriminator network 209' may be adjusted to eliminate detections in images labeled as false positives and / or to produce detections in images labeled as false negatives.

[0104] Figure 3 is a flow chart of an exemplary method 300 for training an object detection network. The method 300 may be performed by one or more processors. Figure 3 Step 301 in: at least one processor may provide a first plurality of images, the first plurality of images comprising a representation of a feature of interest and an indicator of a location of the feature of interest in an image of the first plurality of images. The indicator may comprise a manually set indicator. The manually set indicator may be extracted from a database or received as input from an operator.

[0105] At step 303: at least one processor may train an object detection network using the first plurality of images and the indicator of the feature of interest to detect the feature of interest. For example, the object detection network may be trained as explained above.

[0106] At step 305, the at least one processor may provide a second plurality of images including representations of the feature of interest, the second plurality of images including a greater number of images than included in the first plurality of images. In some embodiments, the second plurality of images may at least partially overlap with the first plurality of images. Optionally, the second plurality of images may consist of different images than the images in the first plurality of images.

[0107] At step 307: at least one processor may apply the trained object detection network to the second plurality of images to produce a first plurality of detections of features of interest. In some embodiments, as explained above, the detections may include indicators of the locations of the detected features of interest. For example, the object detection network may include a convolutional neural network that outputs one or more matrices, each matrix defining the coordinates and / or regions of any detected features of interest, and optionally having one or more associated confidence scores for each detection.

[0108] At step 309: the at least one processor may provide a manually set verification of the true positives and false positives for the first plurality of detection pairs. For example, the at least one processor may extract the manually set verification from a database or receive the verification as input from an operator.

[0109] At step 311: at least one processor may train a generative adversarial network using verification of true positives and false positives with respect to the first plurality of detection pairs. For example, the generative adversarial network may be trained as described above.

[0110] At step 313: at least one processor may retrain the generative adversarial network using at least one additional set of images and detections of features of interest, together with further manually set verification of true positives and false positives regarding the further detections of the features of interest. In some embodiments, the additional set of images may at least partially overlap with the first plurality of images and / or the second plurality of images. Alternatively, the additional set of images may be comprised of images that are different from the first plurality of images and the second plurality of images. Thus, step 313 may include: applying the trained object detection network to the additional set of images to produce further detections of features of interest; providing manually set verification of true positives and false positives regarding the further detections; and retraining the generative adversarial network using the verification regarding the further detections.

[0111] An exemplary method 300 consistent with the present disclosure may include additional steps. For example, in some embodiments, the method 300 may include: the generative network may be retrained by providing a verification of false negatives for missed detected features of interest in two or more images. Thus, the manually set verification extracted from the database or received as input may include a verification of false negatives and a verification of true positives and false positives. The false negatives may be used to retrain the generative adversarial network. Additionally or alternatively, the false negatives may be used to retrain the object detection network.

[0112] Figure 4 4 is a schematic diagram of an object detector 400. The object detector 400 may be implemented by one or more processors. Figure 4As shown, object detector 400 may use database 401 of videos including features of interest. In embodiments where the images include medical images, the features of interest may include abnormalities, such as lesions. Figure 4 In the example of FIG. 4 , database 401 includes a database of endoscopic videos.

[0113] like Figure 4 As shown, detector 400 can extract a subset 403 of videos from database 401. Figure 2 As explained, subset 403 may be selected randomly and / or using one or more patterns. Detector 400 may apply perception branch 407 of discriminator network 405 to frames of subset 403. Perception branch 407 may include an object detection network as described above. Perception branch 407 may be trained to detect features of interest and identify locations (e.g., points or regions) associated with detected features of interest. For example, perception branch 407 may detect anomalies and output a bounding box including the detected anomalies.

[0114] like Figure 4 As shown, perception branch 407 can output detection 413. As explained above, detection 413 can include identifying a point or region of location of a detected feature of interest in subset 403. As further described in Figure 4 As depicted in , detector 400 can extract a subset 411 of videos from database 401. For example, subset 411 can at least partially overlap with subset 403 or be composed of different videos. Subset 411 can have a greater number of videos than subset 403, for example, subset 411 has at least 100 times more videos than subset 403. Detector 400 can use subset 411 and detector 413 to train generative network 415. Generative network 415 can be trained to produce pseudo representations 417 of features of interest, such as anomalies. Pseudo representation 417 can include a false representation of a feature of interest that looks similar to a true representation of the feature of interest. Therefore, generative network 415 can be trained to deceive perception branch 407 into making a detection that is a false positive.

[0115] If further Figure 44, once the generative network 415 is trained, it can generate a pseudo representation 417. The detector 400 can use the pseudo representation 417 to train the adversarial branch 409 of the discriminator network 405. As explained above, the adversarial branch 409 can be trained to distinguish between the pseudo representation 417 and the subset 411. Therefore, the adversarial branch 409 can determine a difference indicator 419. The difference indicator 419 can represent any feature vector or other aspect of the image that is present in the pseudo representation 417 but not in the subset 411, that is present in the subset 411 but not in the pseudo representation 417, or a subtraction vector or other aspect that represents the difference between the feature vector or other aspect of the pseudo representation 417 and its subset 411.

[0116] like Figure 4 As shown, detector 400 can utilize difference indicator 419 to retrain perception branch 407. For example, in embodiments where pseudo representation 417 includes a false representation of a feature of interest, detector 400 can retrain perception branch 407 so that the false representation does not result in the detection of a true representation in subset 411.

[0117] Despite Figure 4 409. Not shown in the figure, the detector 400 can further use recursive training to improve the generative network 415, the perception branch 407, and / or the adversarial branch 409. For example, the detector 400 can use the difference indicator 419 to retrain the generator network 415. Therefore, the output of the adversarial branch 409 can be used to retrain the generator network 415, so that the pseudo representation even looks more similar to the true representation. In addition, the retrained generator network 415 can generate a new set of pseudo representations for retraining the adversarial branch 409. Therefore, the adversarial branch 409 and the generator network 415 can perform unsupervised learning, and the output of each is used to retrain the other outputs in a recursive manner. This recursive training can be repeated until a threshold number of cycles has been reached, and / or until the loss function associated with the generator network 415 and / or the loss function associated with the adversarial branch 409 reaches a threshold. In addition, during this recursive training, each new output of the difference indicator can also be used to retrain the perception branch 407, so that the generator network 415 can be further retrained using a new subset with new detections.

[0118] Figure 5 is a flow chart of an exemplary method 500 for detecting features of interest using a discriminator network and a generator network. The method 500 may be performed by one or more processors.

[0119] exist Figure 5Step 501 in: At least one processor may store a plurality of videos in a database, the videos including representations of features of interest (e.g., abnormalities). For example, the videos may be captured during an endoscopic procedure. As part of step 501, the at least one processor may also select a first subset of the plurality of videos. As explained above, the at least one processor may make the selection randomly and / or using one or more patterns.

[0120] At step 503: at least one processor may apply a perception branch of an object detection network to frames of a first subset of the plurality of videos to produce a first plurality of detections of anomalies. In some embodiments, as explained above, the detections may include indicators of locations of the detected anomalies. Further, in some embodiments, the perception branch may include a convolutional neural network, as explained above.

[0121] At step 505: at least one processor may select a second subset of the plurality of videos. As explained above, the at least one processor may select randomly and / or utilize one or more patterns to select. Using the first plurality of detections and frames from the second subset of the plurality of videos, the at least one processor may further train a generator network to generate a plurality of pseudo representations of anomalies, the plurality of pseudo representations being generated by residual learning. As explained above, each pseudo representation provides a false representation of the anomaly that is highly similar to a true representation of the anomaly.

[0122] At step 507: at least one processor may train an adversarial branch of the discriminator network to distinguish between pseudo representations of anomalies and true representations of anomalies. For example, as explained above, the adversarial branch may be trained to identify the difference between pseudo representations and true representations in a frame. In some embodiments, the adversarial branch may include a convolutional neural network, as explained above.

[0123] At step 509: the at least one processor may apply the adversarial branch of the discriminator network to the plurality of pseudo representations to generate a difference indicator between the pseudo representations of the anomalies included in the frames of the second subset of the plurality of videos and the true representations of the anomalies. For example, as explained above, the difference indicator may represent any feature vector or other aspect of the image that is present in the pseudo representation but not in the frame, that is present in the frame but not in the pseudo representation, or a subtraction vector or other aspect that represents the difference between a feature vector or other aspect of the pseudo representation and its frame.

[0124] At step 511: at least one processor may apply a perception branch of the discriminator network to the pseudo representation to produce a second plurality of detections of anomalies. Similar to the first plurality of detections, the detections may include indicators of locations of detected anomalies in the pseudo representation.

[0125] At step 513: the at least one processor may retrain the perception branch based on the difference indicator and the second plurality of detections. For example, in embodiments where each pseudo-representation provides a false representation of an anomaly that is highly similar to a true representation of the anomaly, the at least one processor may retrain the perception branch to reduce the number of detections returned from the pseudo-representations and, therefore, increase the number of null indicators or other indicators of no anomalies returned from the pseudo-representations.

[0126] An exemplary method 500 consistent with the present disclosure may include additional steps. For example, in some embodiments, the method 500 may include retraining the generative network based on the difference indicator. In this embodiment, the method 500 may further include applying the generative network to generate a plurality of additional pseudo-representations of the anomaly, and retraining the adversarial branch based on the plurality of additional pseudo-representations of the anomaly. This retraining step may be recursive. For example, the method 500 may include applying the retrained adversarial branch to a plurality of additional pseudo-representations to generate a further difference indicator between the further pseudo-representations of the anomaly and the true representation of the anomaly included in the frames of the second subset of the plurality of videos, and retraining the generative network based on the further difference indicator. As explained above, the recursive retraining may be repeated until a threshold number of cycles has been reached and / or until a loss function associated with the generative network and / or a loss function associated with the adversarial branch reaches a threshold.

[0127] Figure 6 6 is a schematic representation of a system 600 including a hardware configuration for a video source consistent with an embodiment of the present disclosure. Figure 6 As shown, system 600 can be communicatively coupled to imaging device 601, such as a camera or other device that outputs a video source. For example, imaging device 601 can include medical imaging equipment, such as a CT scanner, an MRI machine, an endoscopic device, etc. System 600 can also be communicatively coupled to display 615 or other devices for displaying or storing videos. For example, display 615 can include a monitor, screen or other device for displaying images to a user. In some embodiments, display 615 can be replaced or supplemented with a storage device (not shown) or a network interface controller (NIC) that is communicatively connected to a cloud-based storage system (also not shown).

[0128] Further Figure 6 As depicted, system 600 may include an input port 603 for receiving a video source from camera 601, and an output port 611 for outputting video to display 615. As explained above, input port 603 and output port 611 may include a VGA port, an HDMI port, a DVI port, or the like.

[0129] The system 600 also includes a first bus 605 and a second bus 613. Figure 6 As shown, the first bus 605 can transmit the video received through the input port 603 through at least one processor 607. For example, the one or more processors 607 can implement any of the object detector networks and / or discriminator networks described above. Thus, the one or more processors 607 can, for example, use Figure 7 The exemplary method 700 of FIG. 8 superimposes one or more indicators, such as the exemplary graphical indicators of FIG. 8 , on the video received via the first bus 602 . The processor 607 may then send the superimposed video to the output port 611 via the third bus 609 .

[0130] In some cases, the object detector implemented by the one or more processors 607 may fail. For example, the software implementing the object detector may crash or otherwise cease to function properly. Additionally or alternatively, the one or more processors 607 may receive a command (e.g., from an operator of the system 600) to stop overlaying the video. In response to the failure and / or the command, the one or more processors 607 may enable the second bus 613. For example, if Figure 6 As depicted, the one or more processors 607 may send a command or other signal to initiate the second bus 613 .

[0131] like Figure 6 As shown, the second bus 613 can send received video from the input port 603 directly to the output port 611, allowing the system 600 to function as a pass-through for the imaging device 601. The second bus 613 can allow for seamless presentation of video from the imaging device 601 even if the software implemented by the processor 607 fails or if the operator of the hardware overlay 600 decides to stop the overlay in the middle of a video feed.

[0132] Figure 7 is a flow chart of an exemplary method 700 for overlaying object indicators on a video source using an object detector network consistent with an embodiment of the present disclosure. The method 700 may be performed by one or more processors. Figure 7 Step 701: At least one processor may provide at least one image. For example, at least one image may be extracted from a database or received from an imaging device. In some embodiments, the at least one image may include a frame in a video source.

[0133] At step 703: the at least one processor may overlay a border, the border comprising a two-dimensional shape surrounding an area of ​​the image detected as including the feature of interest, the border being rendered as a first color. At step 705: after a period of time has elapsed, if the feature of interest is a true positive, the at least one processor may modify the border to present a second color, and if the feature of interest is a false positive, the border may be modified to present a third color. The elapsed period of time may represent a preset period of time (e.g., a threshold number of frames and / or a number of seconds) and / or may represent the time elapsed between detection of the feature of interest and classification thereof as a true positive or a false positive.

[0134] Additionally or alternatively, if the feature of interest is classified as a first category, the at least one processor may modify the border to a second color, and if the feature of interest is classified as a second category, the at least one processor may modify the border to a third color. For example, if the feature of interest is a lesion, the first category may include cancerous lesions, and the second category may include non-cancerous lesions.

[0135] An exemplary method 700 consistent with the present disclosure may include additional steps. For example, in some embodiments, the method 700 may include sending a command to one or more speakers to produce a sound when the boundary is modified, and / or sending a command to at least one wearable device to vibrate when the boundary is modified. In this embodiment, at least one of the duration, pitch, frequency, and amplitude of the sound and / or vibration may depend on whether at least one detected object is a true positive or a false positive.

[0136] Fig. 8A An exemplary overlay 801 for object detection in a video consistent with an embodiment of the present disclosure is shown. Fig. 8A In the example and Figure 8B and Figure 8C In the example of FIG. 8 , the video samples 800a and 800b are from a colonoscopy procedure. It will be appreciated from this disclosure that videos from other procedures and imaging devices may also be utilized when implementing embodiments of the present disclosure. Thus, the video samples 800a and 800b are non-limiting examples of the present disclosure. Additionally, by way of example, FIG. 8A to FIG. 8C The video display in can be presented on a display device, such as Figure 1 Display 107 or Figure 6 on display 615.

[0137] Overlay 801 represents one example of a graphical border used as an indicator of anomalies or features of interest detected in a video. Fig. 8AAs shown, image 800a and image 800b include frames of a video including detected features of interest. Image 800b includes a graphical overlay 801 and corresponds to a frame that is later in sequence or later in time than image 800a.

[0138] like Fig. 8A As shown, image 800a and image 800b include video frames from a colonoscopy, and the features of interest include lesions or polyps. In other embodiments, as described above, images from other medical procedures (e.g., gastroscopy, colonoscopy, upper gastrointestinal endoscopy (e.g., esophageal endoscopy), etc.) can be utilized and overlaid with a graphical indicator (e.g., overlay 801). In some embodiments, indicator 801 can be overlaid after an abnormality is detected and a certain amount of time has passed (e.g., a certain number of frames and / or seconds between image 800a and image 800b). Fig. 8A In the example of , overlay 801 includes an indicator in the form of a rectangular border having a predetermined pattern (i.e., solid corners). In other embodiments, overlay 801 can be a different shape (whether regular or irregular). In addition, overlay 801 can be displayed in a predetermined color, or transition from a first color to another color.

[0139] exist Fig. 8A In the example of , overlay 801 includes an indicator having a solid corner angle that surrounds the detected location of the feature of interest in the video frame. Overlay 801 appears in video frame 800b, which may be sequentially after video frame 800a.

[0140] Figure 8B Another example of a display with an overlay for object detection in a video is shown in accordance with an embodiment of the present disclosure. Figure 8B An image 810a (similar to image 800a) and a subsequent image 810b (similar to image 800b) superimposed with an indicator 811 are depicted. Figure 8B In the example of FIG. 8 , overlay 811 includes a rectangular border with solid lines on all sides. In other embodiments, overlay 811 can be a first color and / or a different shape (whether regular or irregular). In addition, overlay 811 can be displayed in a predetermined color, or transition from a first color to another color. Figure 8B As shown, overlay 811 is placed on anomalies or features of interest detected in the video. Overlay 811 appears in video frame 810b, which may be sequentially after video frame 810a.

[0141] Figure 8CAnother example of a display with an overlay for object detection in a video is shown in accordance with an embodiment of the present disclosure. Figure 8C An image 820a (similar to image 800a) and a subsequent image 820b (similar to image 800b) superimposed with an indicator 821 are depicted. Figure 8C In the example of FIG. 8 , the overlay 821 includes a rectangular border with dashed lines on all sides. In other embodiments, the indicator 821 can be a different shape (whether regular or irregular). In addition, the overlay 821 can be displayed in a predetermined color, or transition from a first color to another color. Figure 8C As shown, overlay 821 is placed on anomalies or features of interest detected in the video. Overlay 821 appears in video frame 820b, which is sequentially after video frame 820a.

[0142] In some embodiments, the graphical indicator (i.e., overlay 801, 811, or 821) can change pattern and / or color. For example, the pattern and / or color of the border of the pattern can be modified in response to the passage of time (e.g., a specific number of frames and / or seconds between image 800a and image 800b, image 810a and image 810b, or image 820a and image 820b). Additionally or alternatively, the pattern and / or color of the indicator can be modified in response to a specific classification of the feature of interest (e.g., if the feature of interest is a polyp, the polyp is classified as cancerous or non-cancerous, etc.). In addition, the pattern and / or color of the indicator can depend on the classification of the feature of interest. Therefore, if the feature of interest is classified as a first category, the indicator can have a first pattern or a first color, if the feature of interest is classified as a second category, the indicator can have a second pattern or a second color, etc. Alternatively, the pattern and / or color of the indicator can depend on whether the feature of interest is identified as a true positive or a false positive. For example, the feature of interest may be detected by an object detector network as described above (or the perception branch of a discriminator network), thereby generating an indicator, but then determined to be a false positive by an adversarial branch or network as described above, thereby generating an indicator of a first pattern or a first color. Conversely, if the feature of interest is determined to be a true positive by the adversarial branch or network, the indicator may be displayed in a second pattern or a second color.

[0143] The above description is provided for the purpose of illustration. This description is not exhaustive and is not limited to the precise form or embodiment disclosed. By considering the specification and practicing the disclosed embodiments, modifications and adaptations to these embodiments will be apparent. For example, the described implementation includes hardware, but systems and methods consistent with the present disclosure may also be implemented using hardware and software. In addition, although certain components have been described as being coupled to each other, these components may be integrated or distributed with each other in any suitable manner.

[0144] In addition, although illustrative embodiments have been described herein, the scope includes any and all embodiments based on the present disclosure with equivalent elements, modifications, omissions, combinations (e.g., across aspects of the various embodiments), adaptations, and / or changes. The elements in the claims are to be interpreted broadly based on the language used in the claims and are not limited to the examples described in this specification or during the prosecution of the present application (which examples are to be construed as non-exclusive). In addition, the steps of the disclosed methods may be modified in any manner, including rearranging steps and / or inserting or deleting steps.

[0145] The features and advantages of the present disclosure will be apparent from the detailed description, and therefore, the appended claims are intended to cover all systems and methods that fall within the true spirit and scope of the present disclosure. As used herein, the indefinite articles "a" and "an" refer to "one or more". Similarly, the plural terms used do not necessarily mean a plurality unless it is clear in a given context. Unless otherwise specifically indicated, words such as "and" or "or" mean "and / or". In addition, since many modifications and changes will readily occur when studying the present disclosure, it is not desired to limit the present disclosure to the exact structures and operations shown and described, and therefore, all suitable modifications and equivalents may be adopted and fall within the scope of the present disclosure.

[0146] Other embodiments will become apparent from consideration of the specification and practice of the embodiments disclosed herein. It is intended that the specification and examples be considered exemplary only, with a true scope and spirit of the disclosed embodiments being indicated by the following claims.

Claims

1. A system for training a generative adversarial network using an image including a representation of a feature of interest, comprising: at least one memory configured to store instructions; as well as at least one processor configured to execute the instructions to perform operations comprising: providing a first plurality of images, the first plurality of images comprising representations of the features of interest and indicators of locations of the features of interest in images of the first plurality of images; training an object detection network using the first plurality of images and the indicator of the location of the feature of interest to detect the feature of interest; providing a second plurality of images including representations of the features of interest, the second plurality of images including a greater number of images than included in the first plurality of images; applying the trained object detection network to the second plurality of images to produce a first plurality of detections of the features of interest; providing verification of a manual setting of true positives and false positives for said first plurality of detection pairs; training a generative adversarial network using the verification of true positives and false positives for the first plurality of detection pairs; and The generative adversarial network is retrained using at least another set of images and detections of the feature of interest and further manually set verification of true positives and false positives regarding further detections of the feature of interest.

2. The system of claim 1, wherein: The at least one processor is further configured to retrain the generative adversarial network by providing verification of false negatives for missed detection of the feature of interest in two or more images.

3. The system of claim 1, wherein: The object detection network is a convolutional neural network.

4. The system of claim 1, wherein: The number of images in the second plurality of images is at least 100 times greater than the number of images included in the first plurality of images.

5. The system of claim 1, wherein: The first plurality of images and the second plurality of images include medical images.

6. The system of claim 5, wherein: The medical images include images of gastrointestinal organs.

7. The system of claim 1, wherein: At least one of the first plurality of images and the second plurality of images comprises images from an imaging device used during at least one of a gastroscopy, a colonoscopy, or an enteroscopic examination.

8. The system of claim 1, wherein: At least one of the first plurality of images and the second plurality of images includes images from an endoscopic device.

9. The system of claim 1, wherein: The feature of interest is an anomaly.

10. The system of claim 9, wherein: The abnormalities include changes in body tissue from one type of cell to another type of cell.

11. The system of claim 9, wherein: The abnormality includes the absence of body tissue from the location where the body tissue is expected.

12. The system of claim 9, wherein: The abnormality includes: formation on human tissue or formation of human tissue.

13. The system of claim 12, wherein: The abnormality includes a lesion.

14. The system of claim 13, wherein: The lesions include polypoid lesions.

15. The system of claim 13, wherein: The lesions include non-polypoid lesions.

16. The system of claim 1, wherein: The first plurality of detections includes a position indicator when the feature of interest is present or a null indicator when the feature of interest is not present.

17. The system of claim 1, wherein: The at least one processor is further configured to provide false negative verification for images in which the feature of interest is not detected but contains the feature of interest.

18. The system of claim 1, wherein: The indicator of the location of interest comprises a point or an area.

19. The system of claim 1, wherein: The indicators of the locations of interest include manual annotations.

20. The system of claim 19, wherein: The manual annotations include points or regions.

21. The system of claim 1, wherein: The verification of the true positives and the false positives is received from one or more databases or one or more inputs.

Images