Computer-executed system and method for intelligent image analysis using spatio-temporal information

Abstract

Provided is a computer-executed method for detecting at least one feature of interest in an image captured by an imaging device. The method includes receiving an ordered set of images from the captured images, where the ordered set of images is temporally ordered, and analyzing one or more subsets of the ordered set of images using a local spatio-temporal processing module. The local spatio-temporal processing module determines the presence of a characteristic associated with at least one feature of interest in each image of each subset of images and annotates the subset of images based on the determined characteristics of each image of each subset of images. The method further includes processing a set of feature vectors of the ordered set of images using a global spatio-temporal processing module, where the global spatio-temporal processing module is configured to refine the determined characteristics associated with each subset of each image, and calculating one or more values for each image using a time-series analysis module, where the values are calculated using the refined characteristics and spatio-temporal information associated with each subset of the image that represent at least one feature of interest. Additionally, the method may include generating a report, data, or electronic file, integrating with another reporting system or electronic medical record, and / or generating an electronic display of at least one feature of interest using a plurality of values associated with each image of each subset of the ordered set of images.

Description

Technical Field

[0001] The present disclosure generally relates to the field of video processing and image analysis. More specifically, without limitation, the present disclosure relates to systems, methods, and computer-readable media that process video content captured from an imaging device and perform intelligent image analysis such as determining the presence of one or more features of interest or actions performed during a medical procedure. The systems and methods disclosed herein may be used in a variety of applications including medical image analysis and diagnosis.

Background Art

[0002] In video processing and image analysis systems, it is often desirable to detect objects or features of interest. The feature of interest may be a person, place, or thing. In some applications, such as systems and methods for medical image analysis, the location and classification of detected features of interest (e.g., abnormalities such as formations on human tissue or formations of human tissue) are important for patient diagnosis. However, existing computer-executed systems and methods suffer from many drawbacks including the inability to accurately detect features of interest and / or recognize characteristics associated with the features of interest. Additionally, existing systems and methods are inefficient and do not provide a way to intelligently analyze images including those related to the presence of image sequences or events.

[0003] Modern medical procedures require accurate and precise examination of a patient's body and organs. Endoscopic examination is a medical procedure aimed at providing a doctor with video images of the inside of a patient's body and organs for diagnosis. In the digestive tract of the human body, the procedure can be performed by introducing a probe with a video camera through the patient's mouth or anus. During the endoscopic procedure, the doctor manually navigates the probe inside the digestive tract while viewing the video on a display device in real time. The video may be captured, saved, and examined after the endoscopic procedure. As another method, capsule endoscopy is a procedure of swallowing a capsule containing a small camera to examine the patient's digestive tract. The sequence of images taken while the capsule passes through is wirelessly transmitted to a receiving device and saved for the doctor to examine after the procedure. The frame rate of the capsule device varies (e.g., 2 - 6 frames / second), and a large number of images may be taken during the examination procedure.

Summary of the Invention

Problems to be Solved by the Invention

[0004] From the perspective of computer vision, the content captured from either real-time video endoscopy or capsule procedures is a temporally ordered sequence of consecutive images containing information about the patient, e.g., the inner mucosa of the digestive tract. Accurately and precisely analyzing the captured image data is essential for properly examining the patient and identifying lesions, polyps, or other features of interest. Also, typically, the number of images collected for each patient is large. One of the most important medical tasks that a doctor has to perform is to examine this large set of images in order to make an appropriate diagnosis, including determining the presence or absence of features of interest such as pathological regions in the imaged mucosa. However, looking at these images manually is time-consuming and inefficient. As a result, there is a possibility that the doctor may make mistakes and / or misdiagnose during the examination process.

[0005] To improve diagnosis, shorten the time required for medical imaging examinations, and reduce the likelihood of errors, the inventors determined that it would be desirable to have a computer-executable system and method that can intelligently process images and identify the presence of pathology or other features of interest in all images from video endoscopy or capsule procedures or other medical procedures. As an example, features of interest also include actions being performed on or within the image, anatomical or other locations of interest within the image, clinical metric levels of the image, and the like. For this purpose, trained neural networks, spatio-temporal image analysis, and other features and techniques are disclosed herein. As will be understood from this disclosure, the present invention and embodiments can be applied to a wide variety of image capture and analysis applications and are not limited to the examples presented herein.

Means for Solving the Problems

[0006] Embodiments of the present disclosure include a system, a method, and a computer-readable medium that process images captured from an imaging device and perform intelligent image analysis such as determining the presence of one or more features of interest. The systems and methods according to the present disclosure can provide advantages over existing systems and technologies, including addressing one or more of the above and / or other drawbacks of existing systems and technologies. According to some disclosed embodiments, there are provided a system, a method, and a computer-readable medium for processing temporally ordered images from video endoscopy or capsule procedures or other medical procedures. Exemplary embodiments include systems and methods for intelligently processing captured images using spatio-temporal information to accurately assess the likelihood of the presence of abnormalities, pathologies, or other features of interest in the images. As another example, the feature of interest can be a parameter or statistic related to an endoscopy or capsule procedure or other medical procedure. As an example, the feature of interest in an endoscopy procedure may be the clean-out time, i.e., the time it takes for a probe or capsule to traverse through an organ. Also, the feature of interest in an image may be determined based on the presence or absence of characteristics related to the feature of interest. These embodiments, features, and implementations, as well as other embodiments, features, and implementations, are described in more detail herein. The feature of interest may be any feature of or related to one or more images, particularly any feature of or related to the scene or field of view represented in one or more images, which is identifiable or detectable by analyzing the image or each image. The feature of interest may be, for example, an object, a location, an action, or a state (e.g., clinical indicator level).

[0007] In some embodiments, the images captured by an imaging device such as an endoscopic video camera or a capsule camera include images of the gastrointestinal tract or organs. The images may be, for example, from a medical imaging device used during a gastric camera examination, a colon camera examination, or an enteric camera examination. The features of interest in the images may be, for example, abnormalities or other pathologies. An abnormality or pathology may include a formation on or of human tissue, a change from one type of cell to another type of cell in human tissue, an absence of human tissue from where the human tissue is expected, or a formation on or of human tissue. The formation may include a lesion, a polypoid lesion, or a non-polypoid lesion. Other examples of features of interest include anatomical one or other locations, actions, clinical indicators (e.g., cleanliness), etc. As a result, as can be seen from the present disclosure, exemplary embodiments may be medically utilized in a manner that is not specific to any single disease but is generally applicable.

[0008] According to a general aspect of the present disclosure, there is provided a computer-executable system for processing an image captured by an imaging device. The computer-executable system may have at least one processor configured to detect at least one feature of interest in at least one image captured by the imaging device. The at least one processor is configured to receive an ordered set of images from the captured images, the ordered set of images being temporally ordered, and to individually analyze one or more subsets of the ordered set of images using a local spatio-temporal processing module, the local spatio-temporal processing module being configured to determine the presence of a characteristic associated with at least one feature of interest in each image of each subset of images and to annotate the subset images with a feature vector based on the determined characteristics of each image of each subset of images, and to process a set of feature vectors of the ordered set of images using a global spatio-temporal processing module, the global spatio-temporal processing module being configured to refine the determined characteristics associated with each subset of images, each feature vector of the set of feature vectors including information regarding each determined characteristic of the at least one feature of interest, and to calculate a numerical value for each image using a time-series analysis module, the numerical value being configured to represent the presence of the at least one feature of interest and to be calculated using the refined characteristics and spatio-temporal information associated with each subset of images. Further, the at least one processor may be configured to generate a report of the at least one feature of interest using the numerical values associated with each image of each subset of the ordered set of images. The report may be generated after the completion of an endoscopic examination or other medical procedure. The report may include information related to all identified features of interest in the processed images.

[0009] At least one processor of the computer-execution system may be further configured to determine the likelihood of characteristics associated with at least one feature of interest of each image in a subset of images. Further, at least one processor may be configured to encode each image in a subset of images and determine the likelihood of characteristics of each image in the subset of images by aggregating the spatio-temporal information of the determined characteristics using a recurrent neural network or a temporal convolutional network.

[0010] To refine the determined characteristics, a non-causal temporal convolutional network may be utilized. For example, at least one processor of the system may be configured to refine the likelihood of characteristics of each image in a subset of images by applying a non-causal temporal convolutional network. The at least one processor may be further configured to refine the likelihood of characteristics, for example, by applying one or more signal processing techniques including low-pass filtering and / or Gaussian smoothing.

[0011] According to another aspect, at least one processor of the system may be configured to analyze an ordered set of images using a local spatio-temporal processing module to determine the presence of a feature by determining a vector of quality scores, wherein each quality score within the vector of quality scores corresponds to each image of a subset of the images. Further, at least one processor may be configured to process the ordered set of images by using a global spatio-temporal processing module to refine the quality scores of each image of a subset of one or more subsets of the ordered set of images using signal processing techniques. At least one processor may be further configured to analyze one or more subsets of the ordered set of images using a local spatio-temporal processing module to determine the presence of a feature by generating a pixel-wise binary mask for each image of a subset of the images using a deep convolutional neural network. At least one processor may be further configured to process one or more subsets of the ordered set of images using a global spatio-temporal processing module by refining the binary mask for image segmentation using a morphological operation that utilizes prior information regarding the shape and distribution of the determined feature.

[0012] As disclosed herein, the implementation may have one or more of the following features. The likelihood of determining the characteristics of each image in the subset of images may have a floating-point value between 0 and 1. The quality score may be an ordinal number between 0 and R, where score 0 represents the lowest quality and score R represents the highest quality. The numerical value may be associated with each image and may be interpretable to determine the probability of identifying at least one feature of interest within the image. The output may be a first numerical value for an image in which at least one feature of interest was not detected. The output may be a second numerical value for an image in which at least one feature of interest was detected. The size or volume of the subset of images may be settable by a user of the system. The size or volume of the subset of images may be determined dynamically based on the requested features of interest. The size or volume of the subset of images may be determined dynamically based on the determined characteristics. One or more subsets of images may include shared images.

[0013] Another general aspect of the present disclosure relates to a computer-execution system for spatio-temporal analysis of images captured by an imaging device. The computer-execution system may comprise at least one processor configured to receive a video comprising a plurality of image frames captured from the imaging device. The at least one processor accesses a temporally ordered set of images from the captured images, uses an event detection module to detect the occurrence of an event in the temporally ordered set of images, the start time and end time of the event being identified by a start image frame and an end image frame of the temporally ordered set of images, uses a frame selector module to select, from a group of images of the temporally ordered set of images, an image enclosed by the start image frame and the end image frame based on the relevance score and quality score of the image, the relevance score of the selected image representing the presence of at least one feature of interest, uses an object descriptor module to merge a subset of images from the selected images based on the matching presence of at least one feature of interest, identifies the subset of images based on spatial and temporal coherence using spatio-temporal information, and may be configured to divide the temporally ordered set of images at time intervals that satisfy the temporal coherence of the selected task.

[0014] According to the disclosed system, at least one processor may be further configured to determine spatio-temporal information of characteristics related to at least one feature of interest for a subset of images of video content using a local spatio-temporal processing module and to determine spatio-temporal information of all images of the video content using a global spatio-temporal processing module. Further, at least one processor may be configured to divide a temporally ordered set of images into temporal intervals by identifying a subset of the temporally ordered set of images having the presence of at least one feature of interest. Also, at least one processor may be configured to identify a subset of a temporally ordered set of images in which at least one future object of interest exists by adding bookmarks to the images of the temporally ordered set of images, and the bookmarked images are part of a subset of the temporally ordered set of images. Additionally or alternatively, at least one processor may be configured to identify a subset of a temporally ordered set of images in which at least one feature of interest exists by extracting a set of images from a subset of the temporally ordered set of images.

[0015] Embodiments may have one or more of the following features. The extracted set of images may include characteristics related to at least one feature of interest. Color may vary according to the level of relevance of the images of a subset of the temporally ordered set of images to at least one feature of interest. Color may vary according to the level of relevance of the images of a subset of the temporally ordered set of images to characteristics related to at least one feature of interest.

[0016] Another general aspect involves a computer-execution system that performs multiple tasks on a set of images. The computer-execution system may comprise at least one processor configured to receive a video including a set of image frames captured from an imaging device. The at least one processor may be further configured to receive multiple tasks, where at least one of the multiple tasks is associated with a request to identify at least one feature of interest in the set of images, and to analyze a subset of the images in the set of images using a local spatio-temporal processing module to identify the presence of characteristics associated with the at least one feature of interest, and to iterate the execution of a time-series analysis module for each task of the multiple tasks to associate a numerical score for each task with each image in the set of images.

[0017] According to the present disclosure, one or more computer systems can be configured to perform a process or operation by installing software, firmware, hardware, or a combination thereof for the system to execute the process or operation during operation. One or more computer programs can be configured to perform a process or operation by having instructions that cause such an operation or process to be executed by a data processing device (such as one or more processors).

[0018] The systems and methods according to the present disclosure may be implemented using any suitable combination of software, firmware, and hardware. The implementation of the present disclosure may particularly include a program or instructions that are mechanically constructed and / or programmed to perform functions related to the disclosed operations. Additionally, a non-transitory computer-readable storage medium storing program instructions executable by at least one processor to perform the steps and / or methods described herein may be used.

[0019] It is understood that the above general description and the following detailed description are illustrative and explanatory and do not limit the disclosed embodiments.

Brief Description of the Drawings

[0020] The following drawings, which form a part of this specification, illustrate several embodiments and serve to explain, together with the description, the principles and features of the disclosed embodiments.

[0021]

Figure 1A

[0022]

Figure 1B

[0023]

Figure 2

[0024]

Figure 3A

[0025]

Figure 3B

[0026]

Figure 3C

[0027]

Figure 4A

[0028]

Figure 4B

[0029]

Figure 4C

[0030]

Figure 4D

[0031]

Figure 5A

Figure 5B

[0032]

Figure 6A

Figure 6B

[0033]

Figure 6C

Figure 6D

[0034]

Figure 6E

[0035]

Figure 7

[0036]

Figure 8

[0037]

Figure 9

[0038]

Figure 10

[0039] Exemplary embodiments will be described below with reference to the accompanying drawings. The figures are not necessarily drawn to scale. Examples of the disclosed principles and features are described herein, but modifications, adaptations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. Also, the words "comprising," "having," "containing," and "including," and other similar forms, have the same meaning and are unrestricted in that they do not mean an exhaustive listing of such items following any of these phrases nor are they meant to be limited to only the listed items. Also, note that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise.

[0040] In the following description, various examples are provided for illustrative purposes. However, it is understood that the present disclosure can be practiced without one or more of these details.

[0041] Throughout the present disclosure, reference is made to "the disclosed embodiments," which refer to examples of the ideas, concepts, and / or realizations of the invention described herein. Many related and unrelated embodiments are described throughout the present disclosure. The fact that some "disclosed embodiments" are described as exhibiting certain features or characteristics does not mean that other disclosed embodiments necessarily share those features or characteristics.

[0042] Embodiments described herein may refer to a non - transitory computer - readable medium that includes instructions to cause at least one processor to perform a method or a series of operations when executed by the at least one processor. The non - transitory computer - readable medium may be any medium that can store data in any memory in a manner readable by any computer device equipped with a processor for executing a method or any other instructions stored in the memory. The non - transitory computer - readable medium may be implemented as software, firmware, hardware, or any combination thereof. Preferably, the software is implemented as an application program embodied in a program storage unit or a computer - readable medium composed of components, a specific device, and / or a combination of devices. The application program may be uploaded and executed on a machine including any suitable architecture. Preferably, the machine may be realized on a computer platform having hardware such as one or more central processing units (CPUs), a memory, and an input / output interface. The computer platform may include an operating system and microinstruction code. Various processes and functions described in this disclosure may be part of the microinstruction code, part of the application program, or any combination thereof, and may be executed by the CPU regardless of whether such a computer or processor is explicitly shown. Further, various other peripheral units such as additional data storage units and printing units may be connected to the computer platform. Further, the non - transitory computer - readable medium may be any computer - readable medium except a transitory propagation signal.

[0043] The memory may include any mechanism for storing electronic data or instructions, including random access memory (RAM), read-only memory (ROM), hard disk, optical disk, magnetic media, flash memory, other permanent memory, fixed memory, volatile memory, or non-volatile memory. The memory may include one or more separate storage devices, either collocated or distributed, capable of storing data structures, instructions, or any other data. The memory may further include a portion of memory containing instructions for execution by the processor. The memory may be used as the working storage device of the processor or as a temporary storage device.

[0044] Some embodiments may include at least one processor. A processor is any physical device or group of devices equipped with an electrical circuit for performing logical operations on inputs. For example, at least one processor may include one or more integrated circuits (ICs) including application-specific integrated circuits (ASICs), microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), graphics processing units (GPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), servers, virtual servers, or other circuits suitable for executing instructions or performing logical operations. The instructions executed by at least one processor may be preloaded, for example, into memory integrated or embedded in the controller or stored in a separate memory.

[0045] In some embodiments, at least one processor may include multiple processors. Each processor may have a similar configuration, or the processors may have different configurations that are electrically connected or disconnected from each other. For example, the processors may be separate circuits or integrated into a single circuit. When using multiple processors, the processors may be configured to operate independently or cooperatively. The processors may be coupled by electrical, magnetic, optical, acoustic, mechanical, or other means that enable their interaction.

[0046] The disclosed embodiments may include a network. The network may comprise any type of physical or wireless computer networking configuration used for data exchange. For example, the network may be the Internet, a private data network, a virtual private network using a public network, a Wi-Fi® network, a LAN or WAN network, and / or any other suitable connection that enables information exchange between various components of the system. In some embodiments, the network may include one or more physical links used for data exchange, such as Ethernet®, coaxial cable, twisted pair cable, fiber optic cable, or any other suitable physical medium for exchanging data. The network may include a public switched telephone network (PSTN) and / or a wireless cellular network. The network may be a secure network or an insecure network. In other embodiments, one or more components of the system may communicate directly via a dedicated communication network. Direct communication may use any suitable technology, including, for example, BLUETOOTH®, BLUETOOTH® LE (BLE®), Wi-Fi®, near field communication (NFC), or any other suitable communication method that provides a medium for exchanging data and / or information between separate entities.

[0047] In some embodiments, a machine learning network or algorithm may be trained using training examples, for example, as described below. Non-limiting examples of such machine learning algorithms include classification algorithms, data regression algorithms, image segmentation algorithms, visual detection algorithms (such as object detectors, face detectors, person detectors, motion detectors, edge detectors), visual recognition algorithms (such as face recognition, person recognition, object recognition), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbor algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recurrent neural network algorithms, linear machine learning models, non-linear machine learning models, ensemble algorithms, etc. For example, a trained machine learning network or algorithm may include a prediction model, a classification model, a regression model, a clustering model, a segmentation model, an artificial neural network (such as a deep neural network, a convolutional neural network, a recurrent neural network, etc.), a random forest, a support vector machine, etc. In some examples, the training examples may include input examples and desired outputs corresponding to the input examples. Further, in some examples, the training machine learning algorithm using the training examples may generate a trained machine learning algorithm, and the trained machine learning algorithm may be used to estimate outputs for inputs not included in the training examples. Training can be performed with supervision, without supervision, or a combination thereof. In some examples, engineers, scientists, processes, and machines that train machine learning algorithms may further use validation examples and / or test examples.For example, a validation example and / or a test example may include an input example and a desired output corresponding to the input example, and a trained machine learning algorithm and / or an intermediate trained machine learning algorithm may be used to estimate the output of the input example of the validation example and / or the test example, the estimated output may be compared with the corresponding desired output, and the trained machine learning algorithm and / or the intermediate trained machine learning algorithm may be evaluated based on the result of the comparison. In some examples, the machine learning algorithm may have parameters and hyperparameters, the hyperparameters may be set manually by a person or automatically set by a process external to the machine learning algorithm (such as a hyperparameter search algorithm), and the machine learning algorithm is set by the machine learning algorithm according to training examples. In some implementations, the hyperparameters are set according to training examples and validation examples, and the parameters are set according to training examples and the selected hyperparameters. The machine learning network or algorithm may be further retrained based on the output.

[0048] Certain embodiments disclosed herein may include a computer-execution system for performing an operation or method including a series of steps. The computer-execution system and method may be implemented by one or more computer devices that may include one or more processors configured to process real-time video. The computer device may be one or more computers or any other device capable of processing data. Such a computer device may have a display such as an LED display, an augmented reality (AR), or a virtual reality (VR) display. However, the computer device may be implemented in a computing system including back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a user device having a graphical user interface or a web browser through which a user can interact with the implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system and / or computer device may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (“LAN”), wide area networks (“WAN”), and the Internet. The computer device may include clients and servers. Typically, clients and servers are remote from each other and typically communicate through a communication network. The client-server relationship is created by computer programs that are executed on each computer and have a client-server relationship with each other.

[0049] FIG. 1A is a block diagram of an exemplary intelligent detector system 100 according to an embodiment of the present disclosure. As further disclosed herein, the intelligent detector system 100 may be a computer-execution system and may include one or more convolutional neural networks (CNNs) that process images from a medical procedure to identify requested features of interest in the images. The (one or more) features of interest can be a pathology or a list of pathologies that a physician is looking for in an image (e.g., to diagnose a patient). As another example, the features of interest may also include actions performed on or within the image, anatomical locations or other locations of interest within the image, clinical metric levels of the image, and the like. These examples and others are within the scope of the present disclosure. As an example, the action may include an action taken by a physician during or as part of a medical procedure, including an action or procedure identified by the system 100 as a result of a spatio-temporal review of an image from a medical procedure. For example, the procedure may include a recommended procedure or procedure according to medical guidelines such as performing a biopsy, removing a lesion, exploring / analyzing the surface / mucosa of an organ. The action or procedure may be identified based on an image captured and processed by the intelligent detector system 100.

[0050] The intelligent detector system 100 may receive, as input, a set of temporally ordered images of a medical procedure such as an endoscopy or a colonoscopy. The intelligent detector system 100 may output a report or information that includes one or more numerical values (e.g., one or more scores) for each image. The one or more numerical values may be related to a medical category such as a specific pathology and may provide information regarding the probability that the medical category is present within the image frame. The images processed by the intelligent detector system 100 may be images captured from a medical procedure that are stored in a database or storage device for subsequent retrieval and processing by the intelligent detector system 100. In some embodiments, the output provided by the intelligent detector system 100 as a result of processing the images may include, for example, a report having one or more numerical scores assigned to the images and recommended next steps according to one or more medical guidelines. The report may be generated after the completion of the endoscopy or other medical procedure. The report may include information related to all of the features of interest identified in the processed images. Further, in some embodiments, the output provided by the intelligent detector system 100 may include one or more recommended procedures (e.g., performing a biopsy, removing a lesion, exploring / analyzing the surface / mucosa of an organ, etc.) to be performed by a physician in view of the identified one or more features of interest of the images from the medical procedure. During the medical procedure, the intelligent detector system 100 may receive video or image frames directly from a medical imaging device, process the video or image frames, and provide, as feedback to the operator regarding one or more actions performed by the operator during or immediately after the medical procedure (i.e., a short time interval from zero time to a few minutes), and may also provide a final report that includes details regarding a plurality of measured variables, clinical metrics, and what was observed and / or at which anatomical location it was observed and / or how the operator behaved / acted during the medical procedure.The performed operations may include recommended operations or procedures in accordance with medical guidelines such as performing a biopsy, removing a lesion, exploring / analyzing the surface / mucosa of an organ. In some embodiments, the recommended treatment may be part of a set of recommended treatments based on medical guidelines. A detailed description of an exemplary computer system implementing the intelligent detector system 100 for real-time processing is provided in the description of FIG. 1B below.

[0051] As disclosed herein, the intelligent detector system 100 may generate a report after completion of a medical procedure that includes information based on the processing of the captured video by the local spatio-temporal processing module 110, the global spatio-temporal processing module 120, and the time-series analysis module 130. The report may include information related to the features of interest identified during the medical procedure and other information such as the numerical value(s) or score(s) of each image. As described, the numerical value(s) may be related to a medical category such as a specific pathology and may provide information regarding the probability that the medical category is present within the image frame. Further details regarding the operation of the intelligent detector system 100 as well as the local spatio-temporal processing module 110, the global spatio-temporal processing module 120, and the time-series analysis module 130 are provided below with reference to the accompanying drawings.

[0052] In some embodiments, the report generated by the system 100 may include additional recommended operations based on the processing of the saved images from the medical procedure or the real-time processing of the images from the medical procedure. The additional recommended operation(s) may include the operations or procedures performed during the medical procedure and the operations or procedures to be performed after the medical procedure. The additional recommended operation(s) may be part of a set of recommended operation(s) based on medical guidelines. Further, as described above, the system 100 may process the video in real time to provide the operator with simultaneous feedback regarding what is happening or what has been identified in the video and during the medical procedure.

[0053] The output generated by the intelligent detector system 100 may include a dashboard display or a similar report (see, e.g., FIG. 10). The dashboard may provide a summary report of a medical procedure, such as an endoscopy or a colonoscopy. The dashboard may provide a quality score of the procedure and / or other information. The score and / or other information may summarize the examination actions of the medical practitioner and provide information regarding identified features of interest, such as the number of identified polyps. In some embodiments, the information generated by the system 100 is provided and displayed as an overlay on the video from the medical procedure, and thus, the operator can view the information as part of an extended video feed during or immediately after the medical procedure. This information may be provided with some delay or without delay.

[0054] The intelligent detector system 100 may generate reports in the form of electronic files, datasets, or data transmissions. As an example, the output generated by system 100 may conform to a standardized format and / or may be integrated into records such as electronic health records (EHRs). The output of system 100 may comply with regulations such as HIPAA for interoperability and privacy. In some embodiments, the output may be integrated into other reports. For example, the output of the intelligent detector system 100 may be integrated into a patient's electronic medical record or health record. The intelligent detector system 100 may include an API to facilitate such integration and / or may provide the output in the form of a standardized dataset or template. The standardized template may include a predefined form or table that can be filled with data values generated by the intelligent detector system 100 by processing input videos or image frames from medical procedures. In some embodiments, the report may be generated by system 100 in a machine-readable format such as an XML file to support transmission or storage and integration with other systems and applications. In some embodiments, the report may be provided in other formats such as Word®, Excel®, HTML, or PDF file formats. In some embodiments, the intelligent detector system 100 may upload data or reports to a server or database via a network (e.g., refer to network 160 in FIG. 1A). The intelligent detector system 100 may transfer the output data or formatted report to a server or database by making an API call and sending it, for example, as a JSON document.

[0055] Figure 10 shows an exemplary dashboard 1000 having an output summary 1090 generated using an intelligent detector system (such as intelligent detector system 100) according to an embodiment of the present disclosure. The output summary 1090 may be generated for the procedure using the modules of the intelligent detector system 100. The output summary 1090 may provide a quality score and / or other information such as procedure time, extraction time, and clean extraction time. Another example of information that may be part of the summary 1090 includes the time spent performing a particular action (such as one or more of the recommended actions described above) and the time spent at a specific anatomical location. Information related to an interest feature such as a polyp may be provided. For example, the time series analysis module 130 may generate a summary of the number of polyps identified based on characteristics observed in the image frames. The time series analysis module 130 may aggregate information generated by processing the images of the input video using the local spatio-temporal processing module 110 and the global spatio-temporal processing module 120. The summary dashboard 1000 may also include a visual explanation of the interest features identified by the intelligent detector system 100. For example, selected frames of the procedure video may be augmented with markings such as green bounding boxes regarding the location of each identified interest feature, as shown in frames 1092, 1094, and 1096. The frames may relate to different examined parts of the colon such as the ileocecal valve, foramen, and triangular fold, which may themselves be interest features requested by a user of the intelligent detector system 100. The example of FIG. 10 shows information for a procedure presented as part of a single dashboard, but multiple dashboards may be generated with an output summary for each part of the colon or other organ examined as part of a medical procedure. In some embodiments, a combined score or value is generated based on inputs received as a plurality of vectors (e.g., image score vectors) generated by the local spatio-temporal processing module 110 and the global spatio-temporal processing module 120.

[0056] As disclosed above, the features of interest may be related to a medical category or pathology. The intelligent detector system 100 may be implemented to process a request to detect one or more medical categories of an image (i.e., one or more features of interest). In the case of multiple features of interest, one instance of a component of the intelligent detector system 100 may be implemented for each medical category or feature of interest. As will be understood from the present disclosure, an instance of the intelligent detector system 100 may be implemented in any combination of hardware, firmware, and software according to the requirements of speed or throughput, the amount of images to be processed, and other requirements of the system.

[0057] In some embodiments, a single instance of the intelligent detector system 100 may output, for each medical category, a plurality of numerical values for each image, one by one. In one exemplary embodiment, the pathology detected by the intelligent detector system 100 may include the detection of polyps in the colonic mucosa. Further, as an example, the intelligent detector system 100 may output a numerical value (e.g., 0) for all images in the input images in which no polyps were detected by the intelligent detector system 100, and another numerical value (e.g., 1) for all images in the input images in which the intelligent detector detected at least one polyp. In some embodiments, the numerical values can be arranged with respect to a range or scale, and / or the numerical values can indicate the probability of the presence of polyps or other features of interest.

[0058] The source of the input image may vary according to the needs of the imaging device, memory device, and / or application. For example, the intelligent detector system 100 may be configured to process a video feed directly from a video endoscope device according to the embodiments disclosed herein and receive temporally ordered input images that are subsequently processed by the system. As another example, the intelligent detector system 100 may be configured to receive input images from a database or storage device, where the stored images are temporally ordered and were previously captured using an imaging device such as a video camera of an endoscope device or a camera of a capsule device. The images received by the intelligent detector system 100 may be processed and analyzed to identify one or more features of interest such as one or more types of polyps or lesions.

[0059] The exemplary system of FIG. 1A may be implemented in various environments and for various applications. For example, the captured input images may be stored in a local database or storage device, or may be accessed and received by the intelligent detector system 100 via a network from a remote storage location such as cloud storage. Also, the intelligent detector system 100 may be configured to process a streaming video feed from a current medical procedure and process the input images collected from the feed (e.g., via preprocessing and buffering). Further, the operation of the intelligent detector system 100 may be programmed or triggered to start based on one or more conditions. For example, the intelligent detector system 100 may be configured to directly analyze the input images when an input image is received (e.g., via a video feed or a set of stored input images from memory) or when a command is received from a user. The output of the intelligent detector system 100 may be configured as desired. For example, as described above, the intelligent detector system 100 may analyze the input images for one or more features of interest and generate a report indicating the presence of one or more features of interest in the processed images. The report may take the form of an electronic file, a graphical display, and / or an electronic transmission of data. As will be understood, other outputs and report formats are within the scope of the present disclosure. In some embodiments, reports of different formats may be preconfigured and used as templates for report generation by populating the templates with values generated by the intelligent detector system 100. In some embodiments, the report is formatted to be integrated into other reporting systems such as electronic medical records (EMRs). The report format may be a machine-readable format such as XML or Excel® for integration with other reporting systems.

[0060] As an example, the intelligent detector system 100 may process a recorded video or image and provide a fully automated report and / or other output that details the features of interest observed in the processed image. The intelligent detector system 100 may use artificial intelligence or machine learning components to efficiently and accurately process the input image and make a determination about the presence of features of interest based on image analysis and / or spatio-temporal information. Further, for each feature of interest requested or under investigation, the intelligent detector system 100 may estimate its presence within the image and provide a report or other output with information indicating the likelihood of the presence of that feature and other details, such as the procedure or sequence of images in which the feature of interest appears, the relative time from the start of the sequence, the estimated anatomical location, the duration, the most significant images, the location within these images, and / or the number of occurrences.

[0061] In one embodiment, the intelligent detector system 100 may be configured to automatically determine the presence of gastrointestinal pathology without the assistance of a physician. As described above, the input image may be captured and received using various types of imaging devices in various ways. For example, a video endoscope device or a capsule device or other medical device or other imaging device may record and provide the input image. The input image may be part of a live video feed or may be part of a stored set of images received from a local or remote storage location (e.g., a local database or cloud storage). The intelligent detector system 100 may be operated as part of a procedure or service in a clinic or hospital or may be provided as an online service or cloud service for end users to enable self-diagnosis or remote screening.

[0062] As an example, to initiate the inspection procedure, the user may ingest a capsule device or a pill cam. The capsule device may include an imaging device and wirelessly transmit images of the user's gastrointestinal tract during the procedure to a smartphone, tablet, laptop, computer, or other device (e.g., user device 170). The captured images may be uploaded via a network connection to a database, cloud storage, or other storage device (e.g., image source 150). The intelligent detector system 100 may receive input images from the image source and analyze the images for one or more requested features of interest (e.g., polyps or lesions). A final report may be electronically provided as an output to the user and / or their physician. The report may include scoring or probability metrics for each observed feature of interest and / or other relevant information or medical recommendations. Further, the intelligent detector system 100 may detect pathophysiological characteristics associated with the features of interest and serving as indicators thereof, and score those determined to be present. Examples of such characteristics include bleeding, inflammation, ulceration, neoplastic tissue, etc. Further, in response to the detected feature(s) of interest, the report may include information or recommendations based on medical guidelines, such as recommendations to consult a physician and / or undergo additional diagnostic tests. Also, based on the analysis of the images by the intelligent detector system 100, one or more actions (e.g., performing a biopsy, removing a lesion, exploring / analyzing the surface / mucosa of an organ, etc.) may be recommended to the physician in real-time during the medical procedure or after the medical procedure is completed.

[0063] As another example, the intelligent detector system 100 can assist a physician or expert in the analysis of video content recorded during a medical procedure or examination. The captured images can be, for example, part of the video content recorded during a gastric camera examination, a colon camera examination, or an enteric camera examination procedure. Based on the analysis performed by the intelligent detector system 100, a complete video recording can be displayed to the physician or specialist along with a color-coded timeline bar, where different colors correspond to different (one or more) objects of interest and / or scores for the identified (one or more) objects of interest.

[0064] As yet another example, a physician, specialist, or other individual can use the intelligent detector system 100 to create a synopsis of a video recording or image set by focusing on images with desired features of interest and discarding irrelevant image frames. The intelligent detector system 100 can be configured such that a physician or user can adjust or select (one or more) features of interest for detection and the duration of each summary based on total duration and / or other parameters such as preset lead and lag times before and after a sequence of frames having the selected (one or more) features of interest. The intelligent detector system 100 can be configured to combine all frames or the most relevant frames according to the requested (one or more) features of interest.

[0065] As shown in FIG. 1A, the intelligent detector system 100 may include a local spatio-temporal processing module 110, a global spatio-temporal processing module 120, a time series analysis module 130, and a task manager 140. These components may be implemented by any suitable combination of hardware, software, and / or firmware. Further, the number and arrangement of these components may be changed, and it should be understood that the exemplary embodiment of FIG. 1A is provided for illustrative purposes and does not limit the scope of the present invention and its embodiments. Further exemplary features and details related to these components, including with respect to FIGS. 1B and 3A - 3C, are provided below.

[0066] Referring again to the exemplary embodiment of FIG. 1A, the local spatio - temporal processing module 110 may be configured to provide a local perspective by processing (one or more) subsets of images of an input video or a set of input images. The local spatio - temporal processing module 110 may select and process (one or more) subsets of images to generate a score based on a determined presence of characteristics associated with one or more features of interest. For example, assume that the endoscopic input video V includes a set of T image frames. The characteristics may define the features of interest requested by a user of the intelligent detector system 100. For example, the characteristics may include physical and / or biological aspects such as the size, orientation, color, shape, etc. of the feature of interest. The characteristics may include metadata such as data identifying a portion of the video or a period of the video. For example, the characteristics of a video of a colonoscopy procedure may identify (one or more) portions of the colon such as the ascending, transverse, or descending colon. In another example, the characteristics may relate to one or more portions of the video of an endoscopic procedure such as the amount of movement within the image, the presence or movement of an instrument, or the duration of a segment where movement has decreased. The characteristics defining the content may indicate the actions of a physician, clinician, or other individual performing a medical procedure. For example, the portion of the video where the pose with no movement is the longest or the portion of the video where the time spent exploring the surface of an organ is the longest. In some embodiments, the characteristics may be the features of interest. For example, the features of interest and characteristics of a video of a colonoscopy procedure may be a portion of the colon such as the ascending, transverse, or descending colon.

[0067] The local spatio-temporal processing module 110 may be configured to process the entire input video in chunks by iterating over consecutive batches or subsets of N image frames. The local spatio-temporal processing module 110 may be configured to provide an output including a vector or quality score representing the determined characteristics of the (one or more) features of interest in each image frame. In some embodiments, the local spatio-temporal processing module 110 may output a quality value and a segmentation map associated with each image frame. Further exemplary details related to the local spatio-temporal processing module 110 are provided below with reference to the embodiment of FIG. 3A.

[0068] The subset of images processed by the local spatio-temporal processing module 110 may include shared or overlapping images. Further, the size or arrangement of the subset of images may be defined or controlled based on one or more factors. For example, the size or volume of the subset of images may be settable by a physician or other user of the system. As another example, the local spatio-temporal processing module 110 may be configured to dynamically determine the size or volume of the subset of images based on the requested (one or more) features of interest. Additionally or alternatively, the size of the subset of images may be dynamically determined based on the determined characteristics associated with the requested (one or more) features of interest.

[0069] The global spatio-temporal processing module 120 may be configured to provide a global perspective by processing all (one or more) subsets of the image analyzed by the local spatio-temporal processing module 110. For example, the global spatio-temporal processing module 120 may process the entire input video or a set of input images by processing all the outputs of the local spatio-temporal processing module 110 at once or together. Further, the global spatio-temporal processing module 120 may be configured to provide an output including a numerical score for each image frame by processing a vector of determined characteristics related to the (one or more) features of interest. In some embodiments, the global spatio-temporal processing module 120 may process the images and vectors and output a refined quality score and a segmentation map for each image. Further details of additional examples related to the global spatio-temporal processing module 120 are provided below with reference to the embodiment of FIG. 3B.

[0070] The time series analysis module 130 uses information regarding the images determined by the local spatio-temporal processing module 110 and refined by the global spatio-temporal processing module 120 to output a numerical score indicating the presence of one or more characteristics of interest requested by a user of the intelligent detector system 100. For example, the time series analysis module 130 may use the spatial and temporal information of the characteristics related to the (one or more) features of interest determined by the local spatio-temporal processing module 110 to perform a time series analysis on the input video or image. Further exemplary details related to the time series analysis module 130 are provided below with reference to the embodiment of FIG. 3C.

[0071] The task manager 140 may assist in managing various tasks requested by the user of the intelligent detector system 100. The tasks may be related to the requested or necessary features of interest and / or the characteristics of the features of interest. One or more characteristics and features of interest may be part of each task for processing by the intelligent detector system 100. The task manager 140 may assist in task management for the detection of a plurality of features of interest in a set of input images. The task manager 140 may determine the number of instances of the components of the intelligent detector system 100 (e.g., the local spatio-temporal processing module 110, the global spatio-temporal processing module 120, and the time series analysis module 130). Further exemplary details of how to process a plurality of task requests for detecting features of interest are provided below with reference to the descriptions of FIGS. 6 and 7 below.

[0072] The intelligent detector system 100 may receive a set of input videos or images from an image source 150 via a network 160 for processing. In some embodiments, the intelligent detector system 100 may directly receive input videos from another system, such as a medical device or system used to capture videos when performing a medical procedure, e.g., a colonoscopy. After processing the images, a report of the detected features of interest may be shared via the network 160. As disclosed herein, the report may be transmitted electronically, and the report may take various forms such as an electronic file, a display, and data. In some embodiments, the report is transmitted to the user device 170 as a file and / or displayed on the user device 170. The network 160 may take various forms depending on the needs and environment of the system. For example, the network 160 may include the Internet, a wired wide area network (WAN), a wired local area network (LAN), a wireless WAN (e.g., WiMAX (registered trademark)), a wireless LAN (e.g., IEEE 802.11, etc.), a mesh network, a mobile / cellular network, a corporate or private data network, a storage area network, a virtual private network using a public network, and / or any combination of other types of network communications or utilize them. In some embodiments, the network 160 may include an on-premises (e.g., LAN) network, but in other embodiments, the network 160 may include a virtualized, remote, and / or cloud network (e.g., AWS (registered trademark), Azure (registered trademark), IBM Cloud (registered trademark), etc.). Further, the network 160 may, in some embodiments, be a hybrid on-premises and virtualized, remote, and / or cloud network that includes components of one or more types of network architectures.

[0073] The user device 170 may send requests related to the interesting features of the input video or image to the intelligent detector system 100 and receive outputs (e.g., reports or data) from the intelligent detector system 100. The user device 170 may control the input video or image, or directly provide the intelligent detector system 100 with a processing including (one or more) downloads of method instructions, commands, sets of videos or images, and / or (one or more) memory links to storage locations (e.g., the image source 150). The user device 170 may include a smartphone, laptop, tablet, computer, and / or other computer devices. The user device 170 may have an imaging device (e.g., a video camera or digital camera) for capturing videos or images for processing. In the case of a capsule inspection procedure, for example, the user device 170 includes a pillcam or the like that is ingested by the user and captures an input video or image to directly stream to the intelligent detector system 100 or store in the image source 150 and then be downloaded and received by the system 100 via the network 160. The results of the image processing are provided as outputs from the intelligent detector system 100 to the user device 170 via the network 160.

[0074] The physician device 180 may be used to send requests to and receive outputs (e.g., reports or data) from the intelligent detector system 100 related to the features of interest of the input video or image. Similar to the user device 170, the physician device 180 may control the input video or image, or directly provide the intelligent detector system 100 with (one or more) downloads of method instructions, commands, sets of video or image files, and / or (one or more) memory links to storage locations (e.g., the image source 150) for processing. The physician device 180 may comprise a smartphone, laptop, tablet, computer, and / or other computer devices. The physician device 180 may include an imaging device (e.g., a video camera or digital camera) for capturing videos or images for processing. For example, in the case of video endoscopy, the physician device 180 may include a colonoscopy probe having an imaging device for capturing images during a patient's examination. The captured video may be streamed as input video to the intelligent detector system 100 or downloaded and received by the system 100 via the network 160 after being stored in the image source 150. In some embodiments, the physician device 180 may receive notifications for further examination of image frames having features of interest. The results of the image processing are provided as outputs (e.g., electronic reports or data in the form of files or digital displays) from the intelligent detector system 100 to the user device 170 via the network 160.

[0075] The image source 150 may include a storage location or other source for inputting video or images to the intelligent detector system 100. The image source 150 may comprise any suitable combination of hardware, software, and firmware. For example, the image source 150 may include any combination of a computer device, a server, a database, a storage device, network communication hardware, and / or other devices. As an example, the image source 150 may include a database, memory, or storage (e.g., storage 220 in FIG. 2) for storing a set of input videos or images received from the user device 170 and the physician device 180. The storage of the image source 150 may include file storage and / or a database that is accessed using a CPU (e.g., processor 230 in FIG. 2). As another example, the image source 150 may include cloud storage such as AMAZON (registered trademark) S3, Azure (registered trademark) Storage, GOOGLE (registered trademark) Cloud Storage, etc., that is accessible via the network 160.

[0076] In the exemplary system of FIG. 1A, the image source 150, the user device 170, and the physician device 180 may be local to each other or remote from each other and may communicate with each other via wired or wireless communication including network communication. The devices may be local or remote to the intelligent detector network 100 depending on the requirements of the application and system implementation. Further, although the image source 150, the user device 170, and the physician device 180 are shown in FIG. 1A as separate from the intelligent detector system 100, one or more of these devices may be local to the intelligent detector system 100 or provided as part of the system 100. Also, a part or portion of the network 160 may be local to the system 100 or may be part of the system 100. Further, it should be understood that the number and arrangement of the components and devices in FIG. 1A are provided for purposes of illustration and are not intended to limit the present invention or its disclosed embodiments.

[0077] While embodiments of the present disclosure are described herein with general reference to medical image analysis and endoscopy, it should be understood that the embodiments can be applied to other medical image procedures such as gastric camera examination, colon camera examination, and enteric camera examination. Further, embodiments of the present disclosure may be implemented for or in other image capture and analysis environments and systems for LIDAR, surveillance, autonomous driving, and other imaging systems.

[0078] According to one aspect of the present disclosure, there is provided a computer-execution system that intelligently processes a set of input videos or images and determines the presence of and characteristics associated with features of interest. As further disclosed herein, the system (e.g., intelligent detector system 100) may include at least one memory (e.g., ROM, RAM, local memory, network memory, etc.) configured to store instructions and at least one processor (e.g., (one or more) processors 230) configured to execute the instructions (see, e.g., FIGS. 1 and 2). Using at least one processor, the system may process a set of input videos or images captured by a medical imaging system such as used during an endoscopy, gastric camera examination, colon camera examination, or enteric endoscopy procedure. Additionally or alternatively, the image frames may include medical images such as images of gastrointestinal organs or other organs or regions of the human body tissue.

[0079] As used herein, the terms "image frame" or "image" refer to any digital representation of a scene or field of view captured by an imaging device. The digital representation may be encoded in any suitable format such as the Joint Photographic Experts Group (JPEG) format, Graphics Interchange Format (GIF) format, bitmap format, Scalable Vector Graphics (SVG) format, Encapsulated PostScript (EPS) format, and the like. Similarly, the term "video" refers to any digital representation of a scene or region of interest composed of a plurality of consecutive images. The digital representation of the video may be encoded in any suitable format such as the Moving Picture Experts Group (MPEG) format, flash video format, Audio Video Interleave (AVI) format, and the like. In some embodiments, the image sequence of the input video may be paired with audio. As will be appreciated from the present disclosure, embodiments of the present invention are not limited to processing an input video having sequenced or temporally ordered image frames, but may also process a streamed or stored set of sequenced or temporally ordered captured images. Accordingly, the terms "input video" and "(one or more) sets of images" should be considered interchangeable and not limiting of the scope of the present disclosure.

[0080] As disclosed herein, an image frame or image may include a representation of a feature of interest (i.e., an anomaly or other object of interest). For example, the object of interest may include an anomaly on or of human tissue. In other embodiments for non-medical applications, the feature of interest may include an object such as a vehicle, a person, or other entity.

[0081] According to the present disclosure, "abnormalities" include those on human tissue or the formation of human tissue, changes from one type of cell to another type of cell in human tissue, and / or the absence of human tissue from where it is expected to be. For example, the growth of a tumor or other tissue may include an abnormality due to the presence of more cells than expected. Similarly, a bruise or other change in cell type may include an abnormality due to the presence of blood cells in a location other than where they are expected (i.e., outside of a capillary). Similarly, a depression in human tissue may include an abnormality due to the depression being caused as a result of cells not being present where they are expected.

[0082] In some embodiments, the abnormality may include a lesion. The lesion may include a lesion of the gastrointestinal mucosa. The lesion may be classified histologically (e.g., by NICE (Narrow-Band Imaging InternatI / Onal Colorectal Endoscopic) or Vienna classification), morphologically (e.g., by Paris classification), and / or structurally (e.g., as serrated or non-serrated). The Paris classification includes polypoid lesions and non-polypoid lesions. Polypoid lesions may include protruding lesions, lobulated lesions, and protruding or sessile lesions. Non-polypoid lesions may include surface elevation type, flat type, surface shallow depression type, or depression type lesions. Regarding the detection of abnormalities as features of interest, serrated lesions may include sessile serrated adenomas (SSA), traditional serrated adenomas (TSA), hyperplastic polyps (HP), fibroblastic polyps (FP), or mixed polyps (MP). According to the NICE classification system, abnormalities are divided into the following three types. (Type 1) Sessile serrated polyp or hyperplastic polyp, (Type 2) Conventional adenoma, (Type 3) Cancer with deep submucosal invasion. According to the Vienna classification, polyps fall into one of the following five types. (Category 1) Neoplasia / dysplasia negative. (Category 2) Indeterminate for neoplasia / dysplasia. (Category 3) Non-invasive low-grade neoplasms (low-grade adenomas / dysplasia). (Category 4) High-grade adenomas / dysplasia, non-invasive cancer (intraepithelial carcinoma), or high-grade neoplasms of the mucosa such as suspected invasive cancer. (Category 5) Invasive tumors, intramucosal cancer, submucosal cancer, etc. These examples and other types of abnormalities are within the scope of the present disclosure. It should also be understood that the intelligent detector system 100 may be configured to detect other types of features of interest including medical and non-medical procedures.

[0083] Figure 1B is a schematic diagram of an exemplary computer-execution system that implements the intelligent detector system 100 of Figure 1A for processing real-time video according to an embodiment of the present disclosure. As shown in Figure 1B, system 190 includes an image device 192 and an operator 191 that operates and controls the image device 192 through control signals transmitted from the operator 191 to the image device 192. As an example, in an embodiment where the video feed includes medical video, the operator 191 may be a physician or other medical professional. The image device 192 may include a medical imaging device such as an endoscopic imaging device or other medical imaging device that generates video or one or more images of a human body / tissue / organ or a part thereof. The image device 193 may be part of the physician device 180 (as shown in Figure 1A) and generates the video stored in the image source 150. The operator 191 may control the image device 192, in particular, by controlling the capture or frame rate of the image device 192 and / or the movement or navigation of the image device 192, for example, the movement or navigation through or relative to the human body of a patient or individual. In some embodiments, the image device 192 may include a swallowable capsule device or another form of capsule endoscope device, as opposed to an endoscopic imaging device inserted through a cavity of the human body.

[0084] In the example of FIG. 1B, the imaging device 192 may directly transmit the captured video as a plurality of image frames to the computer device 193. The computer device 193 may include a memory (including one or more buffers) and one or more processors for processing video or images, as described above and in this specification (see, e.g., FIG. 2). In some embodiments, one or more of the processors may be implemented as one or more separate components (not shown) that communicate with the computer device 193 but are not part of it, over a network (e.g., network 160 of FIG. 1A). In some embodiments, one or more processors of the computer device 193 may implement one or more networks such as a trained neural network. Examples of neural networks include object detection networks, classification detection networks, position detection networks, size detection networks, or frame quality detection networks, as further described in this specification. The computer device 193 may directly receive and process a plurality of image frames from the imaging device 192. In some embodiments, the computer device 193 may use pre- and / or parallel processing and buffering to process video or images in real time, and the levels of such processing and buffering depend on the frame rate of the received video or image and the processing speed of one or more processors or modules of the computer device 193. As understood, processing and buffering capabilities that are well-matched to the frame rate enable real-time processing and output. Further, in some embodiments, control signals or information signals may be exchanged between the computer device 193 and the operator 191 to control or direct the creation of one or more extended videos as output, and the extended video includes the original video with an overlay (graphics, symbols, text, etc.) that provides information regarding identified features of interest and other feedback generated by the computer device 193 to assist the physician or operator performing the medical procedure.Regarding the control signal or information signal to be exchanged, it may be transmitted as data via the image device 192 or directly transmitted from the operator 191 to the computer device 193. Examples of the control signal and the information signal include signals for controlling components of the computer device 193 such as an object detection network, a classification detection network, a position detection network, a size detection network, or a frame quality detection network as described in this specification.

[0085] In the embodiment of FIG. 1B, the computer device 193 may use one or more modules (such as the modules 110 to 140 of the intelligent detector system 100) to process and expand the video received from the image device 192 and transmit the expanded video to the display device 194. The expanded video can provide, for example, real-time feedback and reports of the procedures taken by the operator 191 during or at the end of a medical procedure such as an identified polyp and an endoscopic examination or a colonoscopy. The expansion or modification of the video may include providing one or more overlays, alphanumeric characters, text, descriptions, shapes, diagrams, images, animation images, and / or other appropriate graphic representations on or with the video frames. The expansion of the video may provide information related to features of interest such as classification, size, performed actions, and / or position information. Additionally or alternatively, the expansion of the video may provide information related to one or more recommended (one or more) actions identified by the computer device 193 in accordance with medical guidelines. It should be understood that the scope and type of information, reports, and data generated by the computer device 193 may be similar to those described above for the intelligent detector system 100 in order to assist the physician or operator and reduce errors. Therefore, refer to the above examples provided for the system 100.

[0086] As further shown in FIG. 1B, the computer device 193 may be configured to directly relay the original unextended video from the image device 192 to the display device 194. For example, the computer device 193 may perform direct relaying under certain conditions such as when there is no overlay or other extension to be generated or when the image device 192 is turned off. In some embodiments, the computer device 193 may perform direct relaying when the operator 191 transmits a command to the computer device 193 as part of a control signal. The command from the operator 191 may be generated by operating buttons and / or keys included in an operator device and / or an input device (not shown) such as a mouse click, cursor hover, mouse over, button press, keyboard input, voice command, interaction performed in virtual reality or augmented reality, or any other input.

[0087] To extend the video, the computer device 193 may create a modified video stream for processing the video from the image device 192 and transmitting it to the display device 194. The modified video may be the original image frame with extension information added to be displayed to the operator via the display device 194. The display device 194 may comprise any suitable display or similar hardware for displaying video or modified video such as an LCD display, an LED display or an OLED display, an augmented reality display or a virtual reality display.

[0088] FIG. 2 shows an exemplary computer device 200 that may be employed in connection with the implementation of the exemplary system of FIG. 1A and other embodiments of the present disclosure. The computer device 200 may be used in connection with the implementation of one or more components of the exemplary system of FIG. 1A (including, for example, the system 100 and the devices 150, 170, and 180). In some embodiments, the computer device 200 may have a plurality of subsystems such as a cloud computing system, a server, and / or any other suitable components for receiving and processing input video and images.

[0089] As shown in FIG. 2, computer device 200 may include one or more processors 230, and the one or more processors 230 may be, for example, all or part of an application specific integrated circuit (ASIC), a microchip, a microcontroller, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a server, a virtual server, or other circuitry suitable for executing instructions or performing logical operations as described above. In some embodiments, the one or more processors 230 may have or be components of a large-scale processing unit implemented with one or more processors. The one or more processors 230 may be implemented in any combination of a general-purpose microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, gated logic, discrete hardware components, a dedicated hardware finite state machine, or any other suitable entity capable of performing calculations or other operations on data or information.

[0090] As further shown in FIG. 2, the one or more processors 230 may be communicatively coupled to memory 240 via a bus or network 250. The bus or network 250 may be adapted to transmit data and other forms of information. Memory 240 may include a memory portion 245 that includes instructions for performing the operations and methods described in detail herein when executed by the one or more processors 230. Memory 240 may optionally be used as the working memory, temporary storage, and other memory or storage of the one or more processors 230. By way of example, memory 240 may be volatile memory such as, but not limited to, random access memory (RAM) or non-volatile memory (NVM) such as, but not limited to, flash memory.

[0091] One or more processors 230 may be communicatively coupled to one or more devices 210 via a bus or network 250. The I / O device 210 may have any type of input device and / or output device or peripheral device, including a keyboard, mouse, display device, etc. The I / O device 210 may have one or more network interface cards, APIs, data ports, and / or other components that support connectivity to one or more processors 230 via the network 250.

[0092] As further shown in FIG. 2, one or more processors 230 and other components (210, 240) of the computer device 200 may be communicatively coupled to a database or storage 220. The storage 220 may electronically store data (e.g., input video or image sets, as well as reports and other output data) in an organized format, structure, or file set. The storage 220 may have a database management system to facilitate storage and retrieval of data. Although the storage 220 is shown in FIG. 2 as a single device, it should be understood that the storage 220 may have multiple databases or storage devices co-located or distributed. In some embodiments, the storage 220 may be implemented in whole or in part as part of a remote network such as cloud storage.

[0093] The processor 230 and / or the memory 240 may have a machine-readable medium that stores software or a set of instructions. As used herein, "software" broadly refers to any type of instruction, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. The instructions may include code (e.g., source code form, binary code form, executable code form, or any other suitable form of code). When executed by one or more processors 230, the instructions may cause the one or more processors to perform various operations and functions described in more detail herein.

[0094] The implementation of the computer device 200 is not limited to the exemplary embodiment shown in FIG. 2. The number and arrangement of the components (210, 220, 230, 240) may be changed and rearranged. Further, although not shown in FIG. 2, the computer device 200 may communicate electronically with other networks including the Internet, local area network, wide area network, metro area network and other networks that can enable communication between elements of the computing architecture. Also, the computer device 200 may search for the data or other information described herein from any source including the storage 220 and (one or more) networks or (one or more) other databases. Further, the computer device 200 may include one or more machine learning models used to implement the neural networks and other modules described herein and may search for or receive the weights or parameters of the machine learning models, training information or training feedback and / or any other data and information described herein.

[0095] FIG. 3A is a block diagram of an exemplary local spatio-temporal processing module according to an embodiment of the present disclosure. The embodiment of FIG. 3A may be used to implement the local spatio-temporal processing module 110 of another computer-executing system such as the exemplary intelligent detector system 100 of FIG. 1A or the computer device 193 of FIG. 1B. As shown in FIG. 3A, the local spatio-temporal processing module 110 has a plurality of components including a sampler 311, an encoder 312, a recurrent neural network (RNN) 313, a temporal convolutional network (TCN) 314, a quality network 315, and a segmentation network 316. These components may be implemented in any suitable combination of hardware, software, and firmware and may be used to select and process a subset of images. The local spatio-temporal processing module 110 may use a batch of N image frames (where N >= 1) as input and repeatedly apply various networks to the entire input video (having T image frames) or image set. Such image frames may be sampled from the input video or image set continuously or at a fixed rate.

[0096] Sampler 311 may select image frames for processing by other components of the local spatio-temporal processing module 110. In some embodiments, Sampler 311 may buffer a set of input videos or images for a set period and extract buffered image frames as a subset of images for processing by module 110. In some embodiments, Sampler 311 may be configured to enable the configuration of the number of frames or size of the image subset selected for processing by the local spatio-temporal processing module 110. For example, Sampler 311 may be configured to receive user input that sets or adjusts the number of frames or size of the image subset for processing. Additionally or alternatively, Sampler 311 may be configured to automatically select the number of image frames or the size of the image subset based on other factors such as the (one or more) features of interest requested for processing and / or the characteristics associated with the (one or more) features of interest. In some embodiments, the amount or size of the sampled images may be based on the frame rate of the video (24, 30, 60, and 120 FPS). For example, Sampler 311 may buffer a real-time video stream received by the intelligent detector system 100 for a set period to extract images from the buffered video. As another example, a stream of images from a pillcam or other imaging device may be buffered, and images may be extracted from the buffer for processing by system 100. In some embodiments, Sampler 311 may selectively sample images based on other factors such as video quality, video length, characteristics associated with the (one or more) features of interest requested, and / or the (one or more) features of interest. In some embodiments, Sampler 311 may sample image frames based on the components of the local spatio-temporal processing module 110 involved in performing the tasks requested by the user of the intelligent detector system 100. For example, an encoder 312 using a 3D encoder network may request multiple images to create a three-dimensional structure of the content to be encoded.

[0097] The encoder 312 may determine the presence of characteristics associated with each feature of interest that is part of the task requested by the user of the intelligent detector system 100. For image analysis, the encoder 312 may be implemented using a trained convolutional neural network. The intelligent detector system 100 may include, as the encoder 312, a 2D encoder and a 3D encoder that include non-local layers. The encoder 312 may be composed of a plurality of convolutional residual layers and fully connected layers. The encoder 312 may select a 2D convolutional encoder or a 3D convolutional encoder according to the detected characteristics and features of interest. The encoder 312 may be trained to detect the characteristics of the image required to detect the requested feature(s) of interest of the image frame. As disclosed herein, the intelligent detector system 100 may process the image and use the encoder 312 to detect the desired characteristics associated with the feature(s) of interest. The intelligent detector system 100 may determine the desired characteristics based on the trained network of the encoder 312 and past determinations of the feature(s) of interest.

[0098] As shown in FIG. 3A, the local spacetime processing module 110 may provide a recurrent neural network (RNN) 313. The RNN 313 may cooperate with the encoder 312 to determine the presence of desirable characteristics related to (one or more) features of interest. In some embodiments, a temporal convolutional network (TCN) 314 may be used to assist in detecting such characteristics in each image frame of the input video. The TCN is a special convolutional neural network that can handle a sequence of images, such as a temporally ordered set of frames of an input video from a source (e.g., the image source 150 of FIG. 1A or the image device 192 of FIG. 1B). The TCN may operate on a subset of the images of all the image frames of the input video buffered for processing by the local processing module 110 or on all the images of the input video processed by the global processing module 120. The TCN can process sequential data using causal convolutions in a fully connected convolutional network.

[0099] The RNN 313 is an artificial neural network having internal feedback connections and an internal memory state used to determine the spacetime information of the image frames. In some embodiments, the RNN 313 may include local layers to improve its ability to aggregate spatial and / or temporally distant spacetime information of the buffered image frames selected by the sampler 311. The RNN 313 can be configured to associate a score, for example, between 0 and 1, with each desirable characteristic related to the requested feature of interest. The score indicates the likelihood of the presence of the desirable characteristic of the image, where 0 represents the lowest likelihood and 1 represents the highest likelihood.

[0100] As shown in FIG. 3A, the local spacetime processing module 110 may have additional components such as a quality network 315 and a segmentation network 316 to further assist in identifying features necessary to detect the features of interest in the processed image. For example, the quality network 315 may be implemented as a neural network composed of a plurality of convolutional layers and a plurality of fully connected layers that improve the final score assigned to the image frame. For example, the quality network 315 may filter image frames having a low feature score. The quality network 315 may generate a feature vector based on the determined characteristics of each image frame. Each feature vector may provide a quality score represented as an ordinal number [0, R] indicating the image quality of the frame, where 0 is the lowest image quality and R is the highest image quality. The quality network 315 may automatically set the quality score 0 for features not detected in the image.

[0101] The segmentation network 316 may process the image to calculate a segmentation mask that extracts a segment of the image containing the characteristics related to the feature of interest for each input image. The segmentation mask may be a pixel-level binary mask having the same or lower resolution as the resolution of the input image. The segmentation network 316 may be implemented as a deep convolutional neural network having a plurality of convolutional residual layers and a plurality of skip connections. The number and type of layers included in the segmentation network may be based on the characteristics or the features of interest required. In some embodiments, a single model having a plurality of layers may handle the tasks of the encoder 312 and the segmentation network 316. For example, the single model can be a U-Net model having a ResNet encoder.

[0102] As an example, the segmentation network 316 may take an image of dimensions W×H as input and return a segmentation mask represented by a matrix of dimensions W’×H’. Here, W’ is less than or equal to W, and H’ is less than or equal to H. Each value of the output matrix represents a probability that includes characteristics related to the feature of interest for which specific image frame coordinates are requested. In some embodiments, the intelligent detector system 100 may generate a plurality of output matrices for a plurality of features of interest. The plurality of matrices may have different dimensions.

[0103] FIG. 3B is a block diagram of an exemplary global spatio-temporal processing module according to an embodiment of the present disclosure. The embodiment of FIG. 3B may be used to implement the global spatio-temporal processing module 120 of another computer execution system such as the exemplary intelligent detector system 100 of FIG. 1A or the computer device 193 of FIG. 1B. The global spatio-temporal processing module 120 may modify or refine the output obtained from the local spatio-temporal processing module 110. Thus, the global spatio-temporal processing module 120 may be configured to process the complete set of images together following the local spatio-temporal processing module 110 processing all subsets of the input video image obtained from the image source 150 via, for example, the network 160. As an example, the global spatio-temporal processing module 120 processes its output to modify the output of the complete set of images processed by the local spatio-temporal processing module 110 by refining and filtering outliers of the images having the determined characteristics or features of interest. In some embodiments, the global spatio-temporal processing module 120 may refine the quality scores and segmentation masks generated by the quality network 315 and the segmentation network 316.

[0104] The global spatio-temporal processing module 120 may refine the scores of the determined characteristics using one or more non-causal temporal convolutional networks (TCNs) 321. The global spatio-temporal processing module 120 may process the outputs of all the images processed by the local spatio-temporal processing module 110 using the dilated convolutional network included as the TCN 321. Such a dilated convolutional network helps increase the receptive field without increasing the depth (number of layers) or kernel size of the network and can be used together for multiple images.

[0105] As further disclosed herein, the TCN 321 may review the entire time series of features K×T’ extracted using the local spatio-temporal processing module 110. The TCN 321 may take as input a matrix of features of dimension K×T’ and may return one or more multiple time series of scalar values of length T”.

[0106] The global spatio-temporal processing module 120 may refine the quality scores generated by the quality network 315 using one or more signal processing techniques such as low-pass filters and Gaussian smoothing. In some embodiments, the global spatio-temporal processing module 120 may refine the segmentation mask generated by the segmentation network 316 using a cascade of morphological operations. The global spatio-temporal processing module 120 may refine the binary mask for segmentation by using the prior information regarding the shape and distribution of the determined characteristics across the input images identified by the encoder 312 in combination with the RNN 313 and the causal TCN 314.

[0107] The TCN321 may operate on a complete set of videos or input images, and thus, it is necessary to wait for the local spatio-temporal processing module 110 to complete the processing of individual image frames. To meet this requirement, the TCN321 may be trained following the training of the network in the local spatio-temporal processing module 110. The number of layers and the architecture of the TCN321 depend on the task(s) required by the user of the intelligent detector system 100 to detect specific (one or more) features of interest. The TCN321 may be trained based on the task(s) required by the system. The training algorithm of the TCN321 may adjust the parameters of the TCN321 for each such task or feature of interest.

[0108] As an example, the intelligent detector system 100 may train the TCN321 by first calculating a K-dimensional time series for each video in the training set 415 using the local spatio-temporal processing module 110 and applying gradient descent-based optimization to estimate the TCN321 parameters to minimize the loss function L(s, s’), where s is the estimated output time series of scores and s’ is the ground truth time series. The intelligent detector system 100 may calculate the distance between s and s’ using, for example, mean squared error (MSE), cross entropy, and / or Huber loss.

[0109] Similar to the training process of other neural networks, data augmentation, learning rate adjustment, label smoothing, and / or batch training may be used when training the TCN321 to improve its capabilities and results.

[0110] The intelligent detector system 100 may be adapted to specific requirements by adjusting the hyperparameters of components 110-130. In some embodiments, the intelligent detector system 100 may change the standard architecture of the pipeline to process a set of input videos or images by adding or removing components 110-130 or specific parts of components 110-130. For example, if the intelligent detector system 100 needs to handle very local tasks, the use of TCN321 in the global spatio-temporal processing module 120 may be reduced to avoid the global spatio-temporal processing of the output generated by the local spatio-temporal processing module 110. As another example, if the user of the intelligent detector system 100 requests a diffusion task, RNN313 and / or TCN314 may be removed from the local spatio-temporal processing module 110. The intelligent detector system 100 may remove the relevant networks from the pipeline used to process the input video or image to detect the required pathology, thereby removing some of the RNN313 and TCN314 in the global spatio-temporal processing module 120 or the local spatio-temporal processing module 110.

[0111] Other arrangements or implementations of the system are also possible. For example, when the required tasks for detecting the features of interest do not handle the lesions in the image frames of the image, the segmentation network 316 may be unnecessary and may be dropped from the pipeline. As another example, when all image frames are considered useful, the quality network 315 may be unnecessary and may be deactivated. For example, when the frame rate of the input video is low or there are many image frames with errors, the intelligent detector system 100 may avoid further filtering the image frames by using the quality network 315. As understood from the present disclosure, the intelligent detector system 100 may preprocess and / or sample the input video or image to determine the components that need to be activated and trained as part of the local spatio-temporal processing module 110 and the global spatio-temporal processing module 120.

[0112] Figure 3C is a block diagram of an exemplary time series analysis module according to an embodiment of the present disclosure. The embodiment of Figure 3C may be used to implement the time series analysis module 130 of other computer-executing systems such as the exemplary intelligent detector system 100 of Figure 1A or the computer device 193 of Figure 1B. The time series analysis module 130 may use, as input, the scores, quality values, and segmentation maps of each image frame processed by the local spatio-temporal processing module 110 to generate the final output score for each image. In some embodiments, after the completion of a medical procedure, the time series analysis module 130 may use the scores, quality values, and / or segmentation maps of the images generated by the local spatio-temporal processing module to generate a dashboard or other display having a summary of the quality scores of all the images. The components of the time series analysis module 130 may be used to generate different output scores and values presented as a summary aggregated for the images processed by the intelligent detector system 100. As shown in Figure 3C, the components of the time series analysis module 130 may include an event detector 331, a frame selector 332, an object descriptor 333, and a time segmenter 334 to assist in generating the final output score for the images of the input video.

[0113] The event detector 331 may determine the start time and stop time of the input video of an event related to the requested feature of interest. In some embodiments, the event detector 331 determines the start image frame and the end image frame of the input video of an event related to the requested feature of interest. In some embodiments, the start time and stop time or the image frames of the event may overlap.

[0114] The start time and stop time of the event may be the start time and end time of the portion of the input video in which some of the characteristics related to the feature of interest are detected. The start time and stop time of the video may be an estimate due to missing image frames from the analysis by the local spatio-temporal processing module 110. The event detector 331 may output a list of pairs (t, d), where t is a time instance and d is a description of the event detected at that time. Various events may be identified based on various features of interest processed by the intelligent detector system 100.

[0115] The portion of the input video identified from the event may include a portion of an organ scanned by a healthcare worker or other operator to generate the input video as part of a medical procedure. For example, a medical procedure such as a colonoscopy may include events configured for various portions of the colon such as the ascending colon, transverse colon, or descending colon.

[0116] The time series analysis module 130 may provide a summary report of the events of various portions of the video representing various portions of the medical procedure. The summary report may include, for example, the time required to complete a scan of a portion of an organ related to an event that may be listed as the length or extraction time of the video. The event detector 331 may assist in generating a summary report of various portions of the medical procedure that include events related to the feature of interest.

[0117] The time series analysis module 130 may present summary reports of (one or more) portions of a medical procedure video (e.g., a colonoscopy video) on a dashboard or other display that shows, for example, a pie chart having amounts of video for portions of the video such as careful secondary review, performing a biopsy, or removing a lesion, or for portions of an organ represented by the video portion that require various actions. In some embodiments, the dashboard may include color-coded quality summary details of events identified by the event detector 331. For example, the dashboard may include red, orange, and green buttons or other icons to identify the quality of a portion of the video representing an event. The dashboard may include summary details of the entire video representing the overall medical procedure at the same level as the information provided for individual portions of the medical procedure.

[0118] In some embodiments, the summary report generated by the time series analysis module 130 may identify one or more frames for further careful review of the portion and / or to address other issues. The summary report may indicate the percentage of the video that requires additional actions such as a second review. The time series analysis module 130 may use the frame selector 332 to identify a particular frame of the video or the percentage of the video that requires additional actions.

[0119] The frame selector 332 may search the image frames of the input video based on the characteristics and scores generated by the local spatio-temporal processing module 110. In some embodiments, the frame selector 332 may utilize quality values provided by the user to select an image frame. The frame selector 332 may select an image frame based on its association with characteristics and / or features of interest requested by a user of the intelligent detector system 100.

[0120] In some embodiments, the summary report generated by the time series analysis module 130 may include one or more image frames identified by the frame selector 332. The image frames presented in the report may be augmented to display (one or more) markings applied to one or more portions of the frames. In some embodiments, the markings may identify features of interest such as lesions or polyps in the image frames. For example, a bounding box colored as a marking surrounding the feature of interest may be used (see, e.g., the green bounding box applied to the image frame shown in FIG. 10). In some embodiments, various markings (including various combinations of (one or more) shapes and / or (one or more) colors) may be used to indicate various features of interest. For example, the image frames may be augmented to include one or more markings in the form of bounding boxes of different colors representing various features of interest identified by the intelligent detector system 100.

[0121] The object descriptor 333 may merge the image frames of the input video that contain matching characteristics from the requested features of interest. The object descriptor merges the image frames based on the temporal coherence and spatial coherence information provided to the local spatio - temporal processing module 110. The output of the object descriptor 333 may include a set of objects described using a set of characteristics. The set of properties may include the time stamps of the image frames relative to other image frames of the input video. In some embodiments, the set of properties may include statistics regarding the estimated scores and positions of the characteristics detected or features of interest requested in the image frames.

[0122] The time segmenter 334 divides the input video into temporal intervals. The time segmenter 334 may perform the division based on the coherence of the task to determine the required features of interest. The time segmenter 334 may output the label of each image frame of the input video in the form of {L_i}. The output label may indicate the presence and probability of the required features of interest of the image frame as well as the position of the image frame. In some embodiments, the time segmenter 334 may output a separate label for each feature of interest of each image frame.

[0123] In some embodiments, the time series analysis module 130 may generate a dashboard or other display that includes a quality score of a medical procedure performed by a physician, healthcare worker, or other operator. To provide the quality score, the time series analysis module 130 may include a machine learning model trained based on videos of medical procedures performed by other physicians and operators having various examination execution actions. In particular, the machine learning model may be trained to recognize video segments in which the examination actions of the healthcare worker indicate a need for additional review. For example, the time during which an endoscopist is carefully exploring the surface of the colon / small intestine may indicate a need for additional review of the small intestine surface, as opposed to the time spent on cleaning or performing surgery or navigation of the colon / small intestine surface. The machine learning model used by the time series analysis module 130 may learn about specific activities of the healthcare worker, such as careful exploration, based on the amount of time spent, the number of photos taken, and / or the number of repeated scans of a specific section of the medical procedure representing a specific part of the organ. In some embodiments, the machine learning model may learn about the actions of the healthcare worker based on the amount of marking in the form of notes or flags added to specific regions of the video or image frames of the video.

[0124] In some embodiments, the time series analysis module 130 may generate a summary report of the quality scores of the actions of medical personnel using information on the time spent performing a particular action (e.g., careful search, navigate, clean, etc.). In some embodiments, the percentage of the total time of a medical procedure for a particular action may be used to calculate the quality score of the medical procedure or a part of the medical procedure. The time series analysis module 130 may be configured to generate a summary report of the quality of the actions of medical personnel based on the configuration of the intelligent detector system 100 so as to include the actions performed by the medical personnel as features of interest.

[0125] To generate a dashboard having the summary scores described above, the time series module may utilize a combination of the event detector 331, the frame selector 332, the object descriptor 333, and the time segmenter 334. The dashboard may include one or more frames from a medical procedure selected by the frame selector 332 and information regarding the total time spent on the medical procedure and the time spent on the examination of parts having pathological features or other features of interest. A summary of the quality scores of the statistics describing the actions of medical personnel may be calculated for the entire medical procedure (e.g., a full colon scan) and / or for parts identifying anatomical regions (e.g., colon segments such as the ascending colon, transverse colon, and descending colon).

[0126] The time series analysis module 130 may use the event detector 331, the frame selector 332, the object descriptor 333, and the time segmenter 334 to generate aggregated information regarding various features of interest such as various regions of the organ captured during a medical procedure, the presence of each pathology, and / or the actions of the medical personnel performing the medical procedure. For example, the aggregated information may be generated based on a list of various pathologies in various regions using the object descriptor 333, the (one or more) frames showing the pathologies selected by the frame selector 332, and the identified amount of time spent in the region of each pathology determined by the event detector 331 and the actions of the medical personnel.

[0127] In some embodiments, the time series analysis module 130 may generate a summary of the input video processed by the local spatio-temporal processing module 110 and the global spatio-temporal processing module 120. The summary of the input video may include segments of the input video that are extracted and combined with the summary of the input video having the features of interest. In some embodiments, the user may select to display only the summary video or expand each section of the video discarded by the module. The time series segmentation 334 of the time series analysis module 130 may assist in extracting portions of the input video having features of interest. In some embodiments, the time series analysis module 130 may generate a video summary by selecting relevant frames to produce a variable frame rate video output. The frame selector 332 of the time series analysis module 130 may assist in the selection and dropping of frames in the output video summary. In some embodiments, the time series analysis module 130 may provide additional metadata to the input video or the video summary. For example, the time series analysis module 130 may color-code the timeline of the input video where features of interest are present. The time series analysis module 130 may use various colors to highlight timelines having various features of interest. In some embodiments, portions of the output video summary having features of interest may include text and graphics overlaid on the output video summary.

[0128] In some embodiments, to maximize performance, modules 110-130 may be trained to select optimal parameter values for the neural networks of each of modules 110-130.

[0129] The components of the local spatio-temporal processing module 110 shown in FIG. 3A may include a neural network that is pre-trained before being used to process images and detect characteristics related to the features of interest. The neural network of the local spatio-temporal processing module 110 may be trained based on a video data set including three subsets: a training set 415, a validation set 416, and a test set 417 (see FIG. 4A). The training subset of the intelligent detector system 100 may require a labeled video set. For example, the labeled video set may include a target score assigned to video processing by the components of the intelligent detector system 100. The labeled video set may include the location of the detectable characteristics in each image frame of the video set and the value of each characteristic. Both the labels and the input video set may be used for training. In some embodiments, the labels of a subset of the video set may be used to determine the labels of other subsets used for training the neural networks of the components 110-130 of the intelligent detector system 100.

[0130] During the training process, the intelligent detector system 100 may sample from a buffer of training dataset images or images processed by the neural networks of the components 110-130 of the intelligent detector system 100 and update their parameters by error backpropagation. The intelligent detector system 110 may use the validation set 416 of the video set to control the convergence of the ground truth value y' of the desired characteristic and the encoder 312 output value y. The intelligent detector system 100 may use the test set 417 of the video set to evaluate the performance of the encoder 312 that determines the value of the characteristic in the image frames of the training subset of the video set. The intelligent detector system 100 may continue training until the ground truth value y' converges to the output value y. The intelligent detector system 100 that has converged may complete the training procedure and temporarily delete the fully connected network completely. The intelligent detector system 100 determines the encoder 312 for the latest values of the parameters.

[0131] FIG. 4A is a flowchart showing an exemplary training of the encoder component of the local spatio-temporal processing module of FIG. 3A according to an embodiment of the present disclosure. An encoder component such as encoder 312 of FIG. 3A takes as input a single image frame or a small buffer of N image frames and generates as output an M-dimensional feature vector. As shown in FIG. 4A, the intelligent detector system 100 may train the encoder 312 by adding a temporary network. The temporary network may be a fully connected network (FCN) 411 added as an additional layer at the end of the encoder 312 to train the encoder 312. The FCN 411 takes as input the feature vector of each image frame of the input generated by the encoder 312 and returns a single floating point value or a one-hot vector y. The intelligent detector system 100 may use a loss function 413 to evaluate the convergence of the ground truth value y' and the output y of the encoder 312. The loss function may be an additional layer added as the last layer of the encoder 312. The loss function 413 may be represented as L(y, y'), which indicates the distance between the ground truth value of the characteristic y' for the image frames of the input video and the output y generated by the encoder 312. The intelligent detector system 100 may use mean squared error (MSE), cross entropy, or Huber loss as the loss function 413 for training the encoder 312 using the FCN 411.

[0132] In some embodiments, the temporary network may be a decoder network 412 used by the intelligent detector system 100 to train the encoder 312. The decoder network 412 may be a convolutional neural network that maps each feature vector estimated by the encoder 312 to a large matrix (I_out) of the same dimension as the image frame (I_in) of the input video. The decoder network 412 may use L(I_out, I_in) as the loss function 413 to calculate the distance between two images (or a buffer of N images). The loss function 413 used with the decoder network 412 may include mean squared error (MSE), structural similarity (SSIM), or L1 norm. The decoder network 412 used as a temporary network for training the encoder 312 does not require the determination of ground truth values for the training / validation / test subsets 415-417 of the video set. The intelligent detector system 100 that trains the encoder 312 using the decoder network 412 as a temporary network may use the validation set 416 to control convergence and the test set 417 to evaluate the expected performance of the encoder 312. The intelligent detector system 110 may stop or deactivate the decoder network 412 after completing the training of the encoder 312.

[0133] In both training methods using the fully connected network 411 and the decoder network 412, the encoder 312 and other components of the intelligent detector system 100 may use techniques such as data augmentation, learning rate tuning, label smoothing, mosaic, MixUp, and CutMix data augmentation and / or batch training to improve the training process of the encoder 312. In some embodiments, the neural network of the intelligent detector system 100 may suffer from class imbalance and may use ad-hoc weighted loss functions and importance sampling to avoid prediction bias for the majority class.

[0134] Figure 4B is a flowchart showing an exemplary training of one or more neural network components of the exemplary local spatio-temporal processing module of Figure 3A. The exemplary training of Figure 4B may be used, for example, to train the recurrent neural network (RNN) 313 and the temporal convolutional network (TCN) 314 of the local spatio-temporal processing module 110 according to embodiments of the present disclosure. The training of deep neural networks (DNNs) in the intelligent detector system 100 such as the RNN 313 and the TCN 314 may require preparing a set of annotated videos or images and a loss function during the training procedure. During the training procedure, the intelligent detector system 100 may adjust the network parameters using a gradient descent-based optimization algorithm.

[0135] The intelligent detector system 100 may train the RNN 313 and the TCN 314 using the output of the previously trained encoder 312. The input to the RNN 313 and the TCN 314 may be the M-dimensional feature vectors for each time instant output by the encoder 312. The RNN 313 and the TCN 314 aggregate a plurality of feature vectors generated by the encoder 312 by buffering the feature vectors generated by the encoder 312. The intelligent detector system 100 may train the RNN 313 and the TCN 314 by supplying a sequence of consecutive image frames to the encoder 312 and passing the generated feature vectors to the RNN 313 and the TCN 314. For a sequence of B images (or a buffered set of images), the encoder 312 generates B vectors of M encoded features and transmits them to the RNN 313 or the TCN 314 to generate B vectors of K features.

[0136] The intelligent detector system 100 may train the RNN 313 and the TCN 314 by including a temporary fully connected network (FCN) 411 at the end of the RNN 313 and the TCN 314. The FCN 411 converts the K-dimensional feature vector generated by the RNN 313 or the TCN 314 into a one-dimensional score and compares it with the ground truth in a loss function to modify the parameters until there is convergence between the output vector and the ground truth vector. In some embodiments, the intelligent detector system 100 improves the RNN 313 and the TCN 314 by using data augmentation, learning rate adjustment, label smoothing, batch training, weighted sampling, and / or importance sampling as part of the training of the RNN 313 and the TCN 314.

[0137] FIG. 4C is a flow diagram showing an example of training the quality network and the segmentation network components of the local spatio-temporal processing module of FIG. 3A. The exemplary training of FIG. 4C may be used to train the quality network 315 and the segmentation network 316 of the exemplary local spatio-temporal processing module 110 in accordance with embodiments of the present disclosure. The intelligent detector system 100 may train the quality network 315 in the same manner as the training of the encoder 312, but does not require a temporary network (FCN 411 or decoder network 412 of FIG. 4A). The quality network 315 outputs a scalar value q representing the quality of the image. The intelligent detector system 100 may train the quality network 315 by comparing its output quality score q with the ground truth quality score q' associated with each image frame of the training set 415 of the video set until the difference in values is minimized. The intelligent detector system 100 may use a loss function 413 represented as L(q,q') to minimize the difference between the ground truth value q' and the output quality score q and to adjust the parameters of the quality network 315. The intelligent detector system 100 may train the quality network 315 using MSE, L1 norm as the loss function 413. The intelligent detector system 100 may use data augmentation, learning rate adjustment, label smoothing, and / or batch training techniques to improve the training results of the quality network 315.

[0138] The intelligent detector system 100 may train the segmentation network 316 using one or more individual images or small buffers of size N. The buffer size N may be based on the number of images considered by the encoder 312 trained in FIG. 4A. The intelligent detector system 100 requires an annotated ground truth map as part of the training set 415 to train the segmentation network 316. The intelligent detector system 100 may use a loss function 413 represented as a loss L(m, m') that defines the distance between the map m estimated by the segmentation network 316 and the ground truth map m'. The intelligent detector system 100 may perform a comparison between the predicted map m and the ground truth map m' by using, for example, pixel-wise MSE and L1 loss functions and dice scores and smooth dice scores.

[0139] In some embodiments, the intelligent detector system 100 may use data augmentation such as ad-hoc morphological operations and affine transformations, learning rate tuning, label smoothing, and / or batch training with each image frame of the input video and the masks generated for each image frame to improve the results of the segmentation network 316.

[0140] FIG. 4D is a flowchart showing a training example of the global spatio-temporal processing module of FIG. 3B. The training example of FIG. 4D may be used to train the global spatio-temporal processing module 120 according to an embodiment of the present disclosure. As shown in FIG. 4D, the intelligent detector system 100 may train the global spatio-temporal processing module 110 by using the output of the local spatio-temporal processing module 110. As understood from the present disclosure, the local spatio-temporal processing module 110 needs to be trained before being used for the training of the global spatio-temporal processing module 120. The local spatio-temporal processing module 110 is trained by training each of its components individually as described in the description of FIGS. 4A-4C above.

[0141] The spatio-temporal convolutional network (TCN) 321 of the global spatio-temporal processing module 120 may access the entire time series T'×K of features extracted by the local spatio-temporal processing module 110 operating on the T' image frames in order to generate a 1×K dimensional feature vector. The global spatio-temporal processing module 120 takes the full matrix T'×K of features as input and returns a time series of scalar values of length T". The intelligent detector system 100 may train the global spatio-temporal processing module 120 by training the TCN 321.

[0142] The intelligent detector system 100 and the global spatio-temporal processing module 120 that train the TCN 321 may consider the number of processing layers of the TCN 321 and its architecture. The number of layers and the connections vary based on the task of determining the features of interest and need to be adjusted for each task.

[0143] The intelligent detector system 100 trains the global spatio-temporal processing module 120 by calculating a K-dimensional time series of scores for the image frames of each video in the training set 415. The intelligent detector system 100 calculates the time series scores by supplying the videos in the training set 415 as input to the previously trained local spatio-temporal processing module 110 and supplying its output to the global spatio-temporal processing module 120. The intelligent detector system 100 may use gradient descent-based optimization to estimate the network parameters of the TCN 321 neural network. The gradient descent-based optimization can minimize the distance between the time series score s output by the global spatio-temporal processing module 120 and the ground truth time series score s'. The loss function 413 used to train the global spatio-temporal processing module 120 can be the mean squared error (MSE), cross entropy, or Huber loss.

[0144] In some embodiments, the intelligent detector system 100 may use data augmentation, learning rate adjustment, label smoothing, and / or batch training techniques to improve the results of the trained global spatio-temporal processing module 120.

[0145] FIGS. 5A and 5B are schematic diagrams of a pipeline constructed of components of an exemplary intelligent detector system for processing an input video or image set. As an example, the pipelines of FIGS. 5A and 5B may be constructed by the components of the intelligent detector system 100 of FIG. 1A that processes an input video. The pipeline for processing an input video using the modules of the intelligent detector system 100 can be structured based on the type of input video to be processed and the required features of interest. The pipeline may include all or some of the components of each module (e.g., the encoder 312 of FIG. 3A).

[0146] As shown in FIG. 5A, the pipeline 500 includes components of the intelligent detector system 100 that process the input video 501 and determine features of interest that may be requested by a user or a physician (e.g., via the user device 170 or the physician device 180 of FIG. 1A). The pipeline 500 includes components of the local spatio-temporal processing module 110 and the global spatio-temporal processing module 120, and generates matrices 531 and 541 of the determined characteristics of each image frame and the scores of the required features of interest. The pipeline 500 also includes a time series analysis module 130 that uses the spatio-temporal information of the characteristics present in the matrices 531 and 541 to determine the features of interest.

[0147] The local spatio-temporal processing module 110 may output a K×T’ matrix 531 of characteristic scores. T’ is the number of frames of the input video 501 repeatedly analyzed by the local spatio-temporal processing module 110. For each analyzed frame of the T’ frames, the local spatio-temporal processing module 110 generates a vector of size K of characteristic scores. The size K may match the number of features of interest requested by the user of the intelligent detector system 100. The local spatio-temporal processing module 110 may process the input video 501 using the sampler 311 to search for some or all of the T image frames. The T’ frames analyzed by the components of the local spatio-temporal processing module to generate the characteristic scores may be less than or equal to all T image frames of the input video 501. The sampler 311 may select the T’ frames for analysis by other components 312 and 315 - 317. In some embodiments, the RNN 313 and the TCN 314 may generate scores only for the T’ image frames of the sampled frames. The networks 313 - 314 may include the T’ image frames based on the presence of at least one characteristic of the requested object of interest. The local spatio-temporal processing module uses only a set of the networks 313 or 314 to process the image frames and generate the matrix 531 of characteristic scores.

[0148] The local spatio-temporal processing module generates a matrix 531 of characteristic scores for the T’ image frames by reviewing each image frame individually or in combination with a subset of the image frames of the input video 501 buffered and provided by the sampler 311.

[0149] The local spacetime processing module 110 may generate additional matrices 532-534 of scores using networks 315-317. The quality network 315 may generate a quality score for each image frame considered by the sampler 311 to determine characteristics related to the features of interest of each image frame. As shown in FIG. 5A, the quality network 315 may generate a matrix 532 of quality scores. The matrix 532 may be a 1×T” vector of quality scores of the T” frames analyzed by the quality network 315. The quality network 315 may analyze the image frames extracted by the sampler 311 to generate quality scores for the T” image frames. In some embodiments, T” may be less than the total number T of frames of the input video 501. The quality network 315 may process only the T” frames having quality scores exceeding a threshold value.

[0150] The segmentation network 316 may generate a matrix 533 of segmentation masks by processing the T''' image frames of the input video 501. The matrix 533 is of dimension W’×H’×T''', and includes T''' masks of height H’ and width W’. In some embodiments, the width W’ and height H’ of the segmentation masks may be smaller than the dimensions of the processed image frames. The segmentation network 316 may analyze the image frames extracted by the sampler 311 to generate segmentation masks for the T''' image frames. In some embodiments, T''' may be less than the total number T of frames of the input video 501. The segmentation network 316 may process the T''' frames with segmentation masks only if they include at least a portion of the characteristics of interest or the required features.

[0151] As shown in FIG. 5A, the global spatio-temporal processing module 120 processes the matrix 531 of K×T’ scores of the features output by the local spatio-temporal processing module 110 to generate a matrix 541 of modified feature scores. The global spatio-temporal processing module 120 reviews the feature scores of all the analyzed T’ image frames by processing the matrix 531 of features together. The TCN321 of the global spatio-temporal processing module 120 may process the matrix 531 of feature scores to generate a matrix 526 of scores of dimension 1×T’. The TCN321 generates the matrix 526 of scores by combining the scores of the T’ image frames represented by vectors of size K. The global spatio-temporal processing module 120 may use a post-processor 522 to remove any outliers within the matrix 526. The post-processor 522 may employ standard signal processing techniques such as low-pass filtering and Gaussian smoothing to remove outliers. The global spatio-temporal processing module 120 outputs a matrix 541 of float scores of dimension 1×U’. The dimension of U’ may be less than or equal to T’. The post-processor 522 may filter a portion of the T’ image frame scores to generate a refined matrix 541 of scores. In some embodiments, the dimension of U’ may be greater than T’ obtained by upsampling the input video in order to increase the number of image frames of the input video (e.g., video 501). The global spatio-temporal processing module 120 may have an upsampling module to increase the number of frames. The global spatio-temporal processing module 120 may upsample the video 501 when the number of image frames having quality scores is less than a threshold. The global spatio-temporal processing module 120 may perform upsampling based on the image frames having high quality scores determined by the quality network 315 of the local spatio-temporal processing module 110. In some embodiments, the video 501 may be upsampled before being processed by the global spatio-temporal processing module 120. For example, the sampler 311 may upsample the input video 501 to create additional image frames.

[0152] The global spatio-temporal processing module 120 may use post-processors 523-525 to refine the additional score and detail matrices 532-534 used when determining the features of interest required to generate matrices 542-544.

[0153] As an example, post-processor 523 refines the quality score matrix 532 using one or more standard signal processing techniques such as a low-pass filter and Gaussian smoothing. Post-processor 523 outputs a matrix 542 of dimension 1×U” of the refined scores. In some embodiments, the value U” may be different from the value T”. For example, if a particular image frame with a low quality score is ignored by post-processor 523, U” may be smaller than T”. Alternatively, if video 501 is upsampled to generate more image frames and image frames with a higher resolution, U” may be larger than T”.

[0154] Post-processor 524 may refine the segmentation mask matrix 533 using a cascade of morphological operations that utilize prior information regarding the shape and distribution of each feature of interest. Post-processor 524 may output a matrix 543 of dimension W’×H’×U'''. In some embodiments, the dimension of U''' may be different from T'''. For example, if a particular image frame with a low quality score is ignored by post-processor 524, U''' may be smaller than T'''. Alternatively, if video 501 is upsampled to generate more image frames and image frames with a higher resolution, U''' may be larger than T'''.

[0155] As shown in FIG. 5A, the time series module 130 of the pipeline 500 may take the output matrices 541-544 from the global spatio-temporal processing module 120 to generate numerical values indicating the positions of the input video of the requested object of interest and the positions of each image frame of the input video. The time series module 130 may use the characteristic scores of the matrix 541 and the quality score matrix 542 to select the image frame that best represents the presence of each feature of interest. In some embodiments, the time series module 130 may utilize the spatio-temporal information of the characteristics of the image frames of the matrix 541 to determine the interval of the input video 501 including the features of interest.

[0156] As shown in FIG. 5B, the pipeline 502 shows an alternative architecture that does not include additional components such as the networks 315-317 (shown in FIG. 5A) and the post-processors 523-525 (shown in FIG. 5A). The pipeline 502 may still generate the same characteristic score matrices 531 and 541 as the outputs of the local spatio-temporal processing module 110 and the global spatio-temporal processing module 120. The time series module 130 takes the matrix 541 as input and generates a value that identifies the features of interest of the input video 501.

[0157] Figures 6A and 6B show various pipeline setups for performing multiple tasks using an intelligent detector system such as the exemplary system 100 of FIG. 1A according to embodiments of the present disclosure. The modules of the intelligent detector system 100 may be configured and managed as a pipeline for processing image data for various tasks. The task manager 140 may maintain various pipeline architectures and manage the data flow across various modules of the pipeline. In some embodiments, the intelligent detector system 100 may be utilized to determine various features of interest requested by various users from the same input video as various tasks. In such a scenario, the neural network of the intelligent detector system 100 may be trained for various tasks to determine the features of interest associated with each task. The intelligent detector system 100 may be trained on various types of input video generated by various medical devices and / or other imaging devices to determine various features of interest.

[0158] The task manager 140 may maintain separate pipelines for each task and train them independently. As shown in FIG. 6A, the intelligent detector system 100 may use a module that generates two separate pipelines 610 and 620 and trains the pipelines 610 and 620 to operate on separate tasks 602 and 603 to process the input video 601 to detect various features of interest. In some embodiments, the pipelines 610 and 630 are pre-trained to handle various tasks. Additionally, the intelligent detector system 100 may instantiate the pipeline by searching for a relevant pre-trained module that processes the input video 601. For example, the task manager 140 may include a local spatio-temporal module 611, a global spatio-temporal module 612, and a time series module 613 in the pipeline 610 to process the video 601 to determine the features of interest required as part of the task 602. Similarly, the pipeline 620 may be constructed using a local spatio-temporal processing module 621, a global spatio-temporal processing module 622, and a time series module 623 to process the video 601 to determine the features of interest as part of the task 603. Maintaining multiple pipelines helps to easily scale to multiple tasks, but may result in redundant processing of image data by certain components. Another efficient aspect of a hybrid pipeline architecture having a partial set of shared components is described in the exemplary embodiment of FIG. 6B below.

[0159] FIG. 6B shows an alternative pipeline 650 that shares modules of the intelligent detector system 100 between tasks. The intelligent detector system 100 shares modules between various tasks by sharing some or all of the components of each module. The intelligent detector system 100 may share these components in a pipeline that has a low dependence on the tasks required to process the image data.

[0160] The sampler 311 and the quality network 315 may act on the image data in the same way regardless of the input image data and the required features of interest. Thus, in the pipeline 650, the component sampler 631 and the quality network 635 of the local spatio-temporal processing module 630, which depend on the input data and are irrelevant to the required tasks, are shared between the tasks 602 and 603 that process the input video 601. The pipeline 650 can share their outputs among multiple tasks processed by components downstream of the pipeline 650.

[0161] The encoder 312 may depend on the required tasks, but may further depend on the input data, and may be shared among different tasks, in order to identify the correct annotation for the image frames of the input video 601. Thus, the pipeline 650 may share the encoder 632 between the tasks 602 and 603. Furthermore, by sharing the encoder 632 among tasks, its training can be improved because the number of available samples across multiple tasks increases.

[0162] The quality network 315 acts directly on the quality of the image without depending on the required tasks. Thus, since the quality score of the image frames of the input video (e.g., the input video 601) is independent of the required tasks (e.g., the tasks 602 and 603) and is the result of applying the same process to the input video 601 multiple times, using separate instances of one quality network 315 for each task would be redundant.

[0163] The segmentation network 316 depends on the required task more than the components described above. However, since it becomes even easier to generate multiple outputs for various tasks (e.g., tasks 602 and 603), sharing can still be performed. As shown in FIG. 6B, the segmentation network 636 is a modified version of the segmentation network 316 and can return multiple segmentation masks for each task as a matrix 653 for each image frame.

[0164] The neural networks 633 - 634 may include either an instance of the RNN 313 or the TCN 314 that generates a matrix of characteristic scores specific to the characteristic of interest required for identification in various tasks. The local spatio - temporal processing module 630 of the pipeline 650 may be configured to generate multiple copies of the encoder outputs 637 and 638 and supply them one by one for each task as inputs to the multiple neural networks 633 and 634.

[0165] FIGS. 6C and 6D show an exemplary pipeline setup for executing multiple tasks with aggregated outputs using an exemplary intelligent detector system according to an embodiment of the present disclosure. As shown in FIG. 6C, the pipelines 610 and 620 generate outputs by simultaneously using multiple time - series analysis modules 671 - 673. For example, the time - series analysis module 673 takes as input the data generated by both of the pipelines 610 and 620. The time - series analysis modules 671 and 672 may generate outputs of an intelligent detector system similar to the outputs of the pipelines 610 and 620 of FIG. 6A described above. The additional time - series analysis module 673 may aggregate the data generated by the local spatio - temporal modules 611, 621 and the global spatio - temporal modules 612 and 621, 622.

[0166] FIG. 6D shows a pipeline that shares the output of the module that performs the time-series analysis and also shares the local space module 630 and the global space module 604. As shown in FIG. 6D, the time-series analysis module 683 takes both the vector 661 that includes the scores of the images generated by preprocessing the images using the sampler 631 and the encoder 632 and the vector 664 that includes the scores of the images generated without preprocessing. The time-series analysis module 683 aggregates the data to generate an output similar to that of the time-series analysis module 673 in FIG. 6C.

[0167] FIG. 6E shows an exemplary dashboard having an output summary of a plurality of tasks generated using an exemplary intelligent detector system according to an embodiment of the present disclosure. The dashboard 690 may provide a summary of a medical procedure performed by a medical professional, such as a colonoscopy. The dashboard 690 may provide information about various parts of the medical procedure and may include scores and / or other information that summarize identified features of interest, such as inspection actions of the medical professional's actions and the number of identified polyps.

[0168] As shown in FIG. 6E, the dashboard 690 may include a summary of the total colon quality score 694 and quality score summaries for various parts of the colon (right colon quality score summary 691, transverse colon quality score summary 692, and left colon quality score summary 693). The quality score summaries 691 - 694 may include time statistics for various actions such as careful exploration, performance of surgery, mucosal washing / cleaning, and rapid movement or navigation of the colon or other human organs. The system 100 may determine, for example, the time to detachment and the amount and / or percentage of time identified as "careful exploration" based on characteristics or factors related to the actions of the medical personnel. The intelligent detector system 100 may identify, for example, the actions performed by the medical personnel as "careful exploration" based on the time spent by the medical personnel analyzing the scanned part of the organ and the time spent analyzing other parts. For example, an endoscopist analyzing the mucosa as opposed to other actions such as washing / removing a lesion may be considered a "careful exploration" action. The time statistics may include summaries of other actions such as the performance of surgery, washing / cleaning of an anatomical region or part of an organ (e.g., mucosa), and rapid movement / navigation of an anatomical location or organ during a medical procedure. Various medical procedures (e.g., colonoscopy, video surgery, video capsule-based scan) may include various actions of the medical personnel as "careful exploration". The intelligent detector system 100 may be configured to label the actions of the medical personnel as "prudent exploration". The quality score summary dashboard 690 may include a color-coded representation of the quality of the examination for each part of the medical procedure. For example, as shown in FIG. 6E, the quality score summary dashboard 690 may include traffic light-colored circles or icons (e.g., red, orange, and green) highlighted to indicate the quality level of the examination for each part of the procedure.

[0169] FIG. 7 is a flowchart showing the operation of an exemplary method for detecting pathology in an input video of an image according to an embodiment of the present disclosure. The steps of method 700 can be performed by the intelligent detector system 100 of FIG. 1A, for example, by executing or using the functions of the computer device 200 of FIG. 2. It is understood that method 700 illustrated can be modified to change the order of steps and include additional steps.

[0170] In step 710, the intelligent detector system 100 may receive an ordered set of input videos or images via network 160. As disclosed herein, the images to be processed may be temporally ordered. The intelligent detector system 100 may request images directly from the image source 150. In some embodiments, other external devices, such as the physician device 180 and the user device 170, may instruct the intelligent detector system 100 to request images from the image source 150. In some embodiments, the user device 170 may submit a request to detect the features of interest of the images currently being streamed or received by the image source 150.

[0171] In step 720, the intelligent detector system 100 may individually analyze a subset of the images to determine the characteristics associated with each requested feature of interest. The intelligent detector system 100 may use the sampler 311 (shown in FIG. 1A) to select the subset of images to be analyzed using other components of the local spatio-temporal processing module 110 (shown in FIG. 1A). Further, as disclosed herein, the local spatio-temporal processing module 110 may limit the subset of images when determining the characteristics of the images.

[0172] The intelligent detector system 100 may enable the configuration of the number of images to include in a subset of images, as disclosed herein. The intelligent detector system 110 may automatically configure the size of the subset based on the requested features of interest or characteristics related thereto. In some embodiments, the user of the intelligent detector system 100 may configure the size of the subset based on input from a user or a physician (e.g., via the user device 170 or the physician device 180 of FIG. 1A). The subsets of images may overlap and share images among them. The intelligent detector system 100 may enable the configuration of the number of overlapping images among subsets of images processed by the local spatio-temporal processing module 110. The intelligent detector system 100 may select a subset of images at one time. In some embodiments, the intelligent detector system 100 may receive a stream of images from the image source 150 and buffer the stream of images until the number of images required to form a subset is reached.

[0173] The intelligent detector system 100 may analyze a subset of images using the local spatio-temporal processing module 110 to determine the likelihood of the characteristics of each image in the subset of images. The likelihood of the features associated with each feature of interest may be represented by a range of continuous or discrete values. For example, the likelihood of a characteristic may be represented using values in the range between 0 and 1.

[0174] The intelligent detector system 100 may detect characteristics by encoding each image of a subset of images using an encoder 312. As part of the analysis process, the intelligent detector system 100 may use a recurrent neural network (e.g., one or more RNNs 313 as shown in FIG. 3A) to aggregate spatio-temporal information of the determined characteristics. In some embodiments, the intelligent detector system 100 may use a causal temporal convolutional network (e.g., one or more TCNs 314 as shown in FIG. 3A) to extract spatio-temporal information of the determined characteristics of each image of the subset of images.

[0175] The intelligent detector system 100 may use a quality network 315 (as shown in FIG. 3A) to determine additional information about each image. The intelligent detector system 100 may use the quality network 315 to determine a vector of quality scores for each image corresponding to the subset of images. The quality scores may be used for relative ranking of each image against an ideal image having the required features of interest. The quality network 315 may output the quality scores as ordinals. The ordinal may be a numerical range where the image needs to be ignored because the quality of the image is very low. For example, the quality network 315 may output quality scores between 0 and R.

[0176] In some embodiments, the intelligent detector system 100 may use a segmentation network 316 to generate additional information about the characteristics. The additional information may include information about portions of the image within each image. The intelligent detector system 100 may use the segmentation network 316 to extract portions of the image having the required features of interest by generating a segmentation mask for each image of the subset of images. The segmentation network 316 may use a deep convolutional neural network to extract the image.

[0177] In step 730, the intelligent detector system 100 may process the image and the vector of information regarding the characteristics of the image determined in step 720. The intelligent detector system 100 may use the global spatio-temporal processing module 120 to process the output generated by the local spatio-temporal processing module 110 in step 720. The intelligent detector system 100 may process the vectors of information related to all the images together to refine the vector of information containing the characteristics determined in each image. The global spatio-temporal processing module 120 may apply a non-causal temporal convolutional network (e.g., the (one or more) temporal convolutional networks (321) of FIG. 3B) to refine the characteristic information generated by the components of the local spatio-temporal processing module 110.

[0178] The intelligent detector system 100 may use a post-processor (e.g., the post-processor 322 as shown in FIG. 3B) to refine the vector with additional information regarding the image and characteristics such as the quality score and the segmentation mask. The intelligent detector system 100 may use one or more signal processing techniques to refine the quality score of each image in the ordered set of images. As an example, the intelligent detector system 100 may use one or more signal processing techniques such as a low-pass filter or Gaussian smoothing to refine the quality score.

[0179] For example, as shown in FIG. 5A, the post-processor 523 may take the quality score matrix 532 of the quality scores to generate a refined score matrix 542.

[0180] In some embodiments, the intelligent detector system 100 may use a post-processor (e.g., post-processor 322 as shown in FIG. 3B) to refine a segmentation mask used for image segmentation to extract portions of each image that contain the requested features of interest. The intelligent detector system 100 may use morphological operations to refine the segmentation mask by leveraging prior information about the shape and distribution of characteristics or features of interest across an ordered set of images. For example, as shown in FIG. 5A, the post-processor 524 may take as input the matrix 533 of the segmentation mask to generate a refined matrix 543 of the segmentation mask.

[0181] In step 740, the intelligent detector system 100 may associate a numerical value with each image based on the refined features of each image in the ordered set of images of step 730. Components of the intelligent detector system 100 may interpret the assigned numerical values of each image to determine the probability of identifying a feature of interest within each image. The intelligent detector system 100 may present various numerical values to indicate various states of each requested feature of interest. For example, the intelligent detector system 100 may output a first numerical value for each image in which the requested feature of interest is detected and a second numerical value for each image in which the requested feature of interest is not detected.

[0182] In some embodiments, the intelligent detector system 100 may interpret the associated numerical values to determine the number of images containing the location or characteristics of the requested feature of interest. Following step 740, the intelligent detector system 100 may generate a report containing information about each feature of interest based on the numerical values associated with each image (step 750). As disclosed above, the report may be electronically presented in various forms (e.g., file, display, data transmission, etc.), and the report may include not only information regarding the presence of each requested feature of interest, but also, for example, additional information and / or recommendations based on medical guidelines. Upon completion of step 750, the intelligent detector system 100 completes, for example, the process at the computer device 200 (step 799) and the execution of method 700.

[0183] FIG. 8 is a flowchart showing the operation of an exemplary method for spatio-temporal analysis of video content according to an embodiment of the present disclosure. The steps of method 800 may be performed by the intelligent detector system 100 of FIG. 1A, for example, by executing or using the functions of the computer device 200 of FIG. 2. It is understood that the illustrated method 900 may be modified to change the order of the steps and include additional steps.

[0184] In step 810, the intelligent detector system 100 may access a temporally ordered set of images of video content from the image source 150 (shown in FIG. 1A) via the network 160 (shown in FIG. 1A). In some embodiments, the intelligent detector system 100 may access the images by extracting the images from the input video. In some embodiments, the received images may be stored in and accessed from memory.

[0185] In step 820, the intelligent detector system 100 may detect the occurrence of events in a temporally ordered set of images using the spatio-temporal information of the features of each image in the ordered set of images. The intelligent detector system 100 may detect events using an event detector 331 (as shown in FIG. 3C). The intelligent detector system 100 may determine spatio-temporal information using a local spatio-temporal processing module 110 and a global spatio-temporal processing module 120. The intelligent detector system 100 may determine spatio-temporal information in a two-step method. First, the local spatio-temporal processing module may search for spatio-temporal information regarding characteristics by reviewing each image in the accessed set of images. In some embodiments, the local spatio-temporal processing module 110 may use a subset of the images. Next, the global spatio-temporal processing module 120 may use the spatio-temporal information regarding the characteristics local to each image to generate combined spatio-temporal information for all the images by reviewing the spatio-temporal information of all the images generated by the local spatio-temporal processing module 110.

[0186] When detecting an event, the intelligent detector system 100 may add color to a portion of the timeline of the video content that corresponds to a subset of the temporally ordered set of images of the video content in which the event was discovered.

[0187] The color may vary according to the level of association of the images in the subset of the temporally ordered set of images with respect to the characteristics related to the feature of interest. The color may vary according to the level of association of the images in the subset of the temporally ordered set of images with respect to one or more characteristics.

[0188] The intelligent detector system 100 may use the determined spatio-temporal information of the characteristics to make a determination among the temporally ordered set of images in which there is an event representing the occurrence of the feature of interest.

[0189] In step 830, the intelligent detector system 100 may select an image from a group of images using a frame selector 332 (as shown based on FIG. 2) based on an associated score and a quality score of the image indicating the presence of characteristics related to at least one feature of interest. The intelligent detector system 100 may use a quality network 335 to evaluate the quality score of each image of the images accessed in step 810. The frame selector 332 may use the quality score generated by the quality network 335 and the feature score generated in step 820 to review the images and determine the images with information. The intelligent detector system 100 may select an image frame by adding bookmarks to a temporally ordered set of images.

[0190] In step 840, the intelligent detector system 100 may use an object descriptor 333 to merge a subset of images having matching characteristics based on spatial and temporal coherence. The intelligent detector system may use the spatio-temporal information of the characteristics of each image determined in step 820 to determine the spatio-temporal coherence of the characteristics.

[0191] In step 850, the intelligent detector system 100 may use a temporal segmenter 334 (as shown in FIG. 3C) to divide a temporally ordered set of images that meet the temporal coherence of the selected task. The intelligent detector system 100 may divide a set of images by identifying a subset in which one or more features of interest are present. The intelligent detector system 100 may use the spatio-temporal information of the features determined in step 820 to determine the temporal coherence. The intelligent detector system 100 may consider that an image has temporal coherence if the presence of one or more features of interest matches.

[0192] The intelligent detector system 100 may extract a clip of video content that matches one of the divided subsets of a temporally ordered set of video images. The extracted clip may contain at least one relevant feature. When step 850 is completed, the intelligent detector system 100 completes executing 800 on the computer device 200 (step 899).

[0193] FIG. 9 is a flowchart showing the operations of an exemplary method 900 for a plurality of tasks on a set of input images according to an embodiment of the present disclosure. The steps of method 900 may be executed by the intelligent detector system 100 of FIG. 1A, for example, by executing or using the functions of the computer device 200 of FIG. 2. It is understood that method 900 illustrated may be modified to change the order of steps and include additional steps.

[0194] In step 910, the intelligent detector system 100 may receive an input video (e.g., input video 601 of FIG. 6A) including a plurality of tasks (e.g., tasks 602 and 603 of FIG. 6A) and a set of images. Each of the received tasks 602 and 603 may include a request to identify relevant features of a set of input images of the input video.

[0195] In step 920, the intelligent detector system 100 may analyze a subset of images using the local spatio-temporal processing module 110 (shown in FIG. 1A) to identify the presence of characteristics associated with each requested relevant feature of each image in the subset of images.

[0196] In some embodiments, the intelligent detector system 100 may use the global spatio-temporal analysis module 120 to refine the features identified by the local spatio-temporal processing module 110 by filtering out mis-identified features. In some embodiments, the global spatio-temporal processing module 120 may highlight and flag some of the features identified by the local spatio-temporal processing module 110. In some embodiments, the global spatio-temporal processing module 120 may perform filtering using additional components such as the quality network 315 and the segmentation network 316 that are applied once to a set of images to generate additional information about the input image.

[0197] In step 930, the intelligent detector system 100 may repeatedly execute the time-series analysis module 130 for each task of the requested set of tasks to associate a numerical score with each image of the input set of images. In some embodiments, the intelligent detector system 100 may include multiple instances of the time-series module 130 to process multiple tasks simultaneously. For example, the time-series modules 671 and 672 (as shown in FIG. 6B) simultaneously identify different sets of features of the same set of images for different tasks 602 and 603. The intelligent detector system 100 completes executing 900 on the computer device 200 upon completion of step 930 (step 999).

[0198] The figures and components of the drawings described above illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer hardware or software products according to various exemplary embodiments of the present disclosure. For example, each block of a flowchart or diagram may represent a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function. It should also be understood that in some alternative embodiments, the functions represented by the blocks may occur out of the order described in the figures. As an example, two blocks or steps shown consecutively may be executed or implemented substantially simultaneously, or the two blocks or steps may sometimes be executed in the reverse order depending on the related functions. Additionally, some blocks or steps may be omitted. Furthermore, it should be understood that each block or step of the figures and combinations of blocks or steps may be implemented by a special-purpose hardware-based system for performing the specified function or operation or by a combination of special-purpose hardware and computer instructions. A computer program product (e.g., software or program instructions) may be realized based on the described embodiments and illustrated examples.

[0199] It should be understood that the systems and methods described above can be modified in many ways and the various features can be combined in various ways. In particular, not all of the features shown above are necessary in all embodiments or implementations. Other combinations of the above features and embodiments are also considered to be within the scope of the embodiments or implementations disclosed herein.

[0200] In this specification, specific embodiments and implementation features have been described and illustrated, but variations, substitutions, modifications, and equivalents will be apparent to those skilled in the art. Accordingly, it is to be understood that the appended claims are intended to cover all such variations and modifications that fall within the scope of the disclosed embodiments and the illustrated implementation features. Also, it is to be understood that the embodiments described in this specification are presented by way of example only and not by way of limitation, and various modifications in form and detail may be made. Any part of the systems and / or methods described in this specification may be implemented in any combination, except combinations that are mutually exclusive. By way of example, the embodiments described in this specification may include various combinations and / or sub-combinations of the functions, components, and / or features of the various embodiments described.

[0201] Furthermore, exemplary embodiments have been described herein, but the scope of the disclosure includes embodiments having equivalent elements, variations, omissions, combinations (e.g., combinations of aspects across various embodiments), adaptations, or replacements based on the embodiments disclosed herein. Further, the elements of the claims should be construed broadly based on the language employed in the claims and not limited to the examples described in this specification or during the prosecution of this application. Rather, these examples are to be construed as non-exclusive. Accordingly, this specification and the examples are intended to be considered by way of example only, and the true scope and spirit are indicated by the full scope of the following claims and their equivalents.

Claims

1. A computer execution system for processing images, One or more memory devices that store processor-executable instructions, One or more processors, configured to execute instructions causing the one or more processors to perform steps for performing a set of images, wherein the steps include: Receiving the aforementioned plurality of tasks, wherein at least one of the plurality of tasks is associated with a request to identify at least one feature of interest of the set of images, To identify the presence of a feature associated with at least one feature of interest, a local spatiotemporal processing module is used to analyze a subset of images from the set of images, In order to associate the numerical score of each task with each image in the subset of images, the execution of the time series analysis module is repeated for each of the multiple tasks. One or more processors equipped with, A system equipped with these features.

2. The system according to claim 1, wherein the local spatiotemporal processing module outputs an analyzed subset of the set of images, and each subset is associated with a task of the plurality of tasks.

3. The system according to claim 1, wherein the local spatiotemporal processing module determines the presence of the characteristic by determining a vector of quality scores, and each quality score in the vector of quality scores corresponds to each image in a subset of the images.

4. The system according to claim 3, wherein each quality score is an ordinal number from 0 to R, with a score of 0 representing the lowest quality and a score of R representing the highest quality.

5. The system according to claim 1, wherein the local spatiotemporal processing module generates a set of feature vectors of features of interest related to the plurality of tasks.

6. The system according to claim 1, further comprising using a global spatiotemporal processing module to analyze a set of feature vectors of a subset of the images analyzed by the local spatiotemporal processing module.

7. The system according to claim 1, further comprising using the time series analysis module to aggregate the outputs of the local spatiotemporal processing module for each of the plurality of tasks.

8. The system according to claim 1, wherein the set of images is received directly from an imaging device during a medical procedure.

9. The system according to any one of claims 1 to 8, which determines the presence of at least one feature of interest from a portion of a captured image.

10. A non-temporary computer-readable medium having instructions that cause at least one processor to perform steps for performing a set of images when executed by at least one processor, wherein the steps are: Receiving the aforementioned plurality of tasks, wherein at least one of the plurality of tasks is associated with a request to identify at least one feature of interest of the set of images, To identify the presence of a feature associated with at least one feature of interest, a local spatiotemporal processing module is used to analyze a subset of images from the set of images, In order to associate the numerical score of each task with each image in the subset of images, the execution of the time series analysis module is repeated for each of the multiple tasks. A computer-readable medium that includes [a certain feature].

11. The computer-readable medium according to claim 10, wherein the local spatiotemporal processing module outputs an analyzed subset of the set of images, each subset being associated with a task of the plurality of tasks.

12. The computer-readable medium according to claim 10, wherein the local spatiotemporal processing module determines the presence of the characteristic by determining a vector of quality scores, and each quality score in the vector of quality scores corresponds to each image in a subset of the images.

13. The computer-readable medium according to claim 12, wherein each quality score is an ordinal number from 0 to R, where a score of 0 represents the lowest quality and a score of R represents the highest quality.

14. The computer-readable medium according to claim 10, wherein the local spatiotemporal processing module generates a set of feature vectors of features of interest related to each of the plurality of tasks.

15. The computer-readable medium according to claim 10, further comprising the step of analyzing a set of feature vectors of a subset of the image analyzed by the local spatiotemporal processing module using a global spatiotemporal processing module.

16. The computer-readable medium according to any one of claims 10 to 15, further comprising using the time-series analysis module to aggregate the outputs of the local spatiotemporal processing module for each of the plurality of tasks.

17. A computer execution method for performing multiple tasks on a set of input images, comprising a step performed by at least one processor, wherein the step is: Receiving the aforementioned plurality of tasks, wherein at least one of the plurality of tasks is associated with a request to identify at least one feature of interest of a set of images, To identify the presence of a feature associated with at least one feature of interest, a local spatiotemporal processing module is used to analyze a subset of images from the set of images, In order to associate the numerical score of each task with each image in the subset of images, the execution of the time series analysis module is repeated for each of the multiple tasks. A method for providing this.

18. The method according to claim 17, wherein the local spatiotemporal processing module outputs an analyzed subset of the set of images, each subset being associated with a task of the plurality of tasks.

19. The method according to claim 17, wherein the local spatiotemporal processing module determines the presence of the characteristic by determining a vector of quality scores, and each quality score in the vector of quality scores corresponds to each image in a subset of the images.

20. The method according to claim 19, wherein each quality score is an ordinal number from 0 to R, where a score of 0 represents the lowest quality and a score of R represents the highest quality.

21. The method according to claim 17, wherein the local spatiotemporal processing module generates a set of feature vectors of features of interest related to each of the plurality of tasks.

22. The method according to claim 17, further comprising using a global spatiotemporal processing module to analyze a set of feature vectors of a subset of the images analyzed by the local spatiotemporal processing module.

23. The method according to claim 17, further comprising using the time series analysis module to aggregate the output of the local spatiotemporal processing module for each of the plurality of tasks.

24. The method according to claim 17, wherein the ordered set of images is received directly from an imaging device during a medical procedure.

25. The method according to any one of claims 17 to 24, wherein the presence of at least one feature of interest is determined from a portion of a captured image.

Images