Computer-implemented system and method for intelligent image analysis using spatio-temporal information

Abstract

A computer-implemented method is provided for detecting at least one feature of interest in an image captured with an imaging device. The method includes receiving an ordered set of images from the captured images, the ordered set of images being temporally ordered, and analyzing one or more subsets of the ordered set of images using a local spatiotemporal processing module, the local spatiotemporal processing module is configured to determine a presence of a characteristic related to at least one feature of interest in each image of each subset of images, and annotate the subset of images based on the determined characteristic in each image of each subset of images. The method further includes processing the set of feature vectors of the ordered set of images using a global spatiotemporal processing module configured to refine the determined characteristics associated with each subset of images, and calculating one or more values for each image using a time series analysis module, the numerical value represents at least one feature of interest and is calculated using refined features associated with each image subset and spatio-temporal information. Yet further, the method may include generating a report, data, or electronic file using the plurality of values associated with each image of each subset of the ordered set of images, integrating into another reporting system or electronic medical record, and / or generating an electronic display on at least one feature of interest.

Description

Technical Field

[0001] The present disclosure relates generally to the fields of video processing and image analysis. More specifically, and without limitation thereto, the present disclosure relates to systems, methods, and computer-readable media for processing captured video content from an imaging device and performing intelligent image analysis (e.g., determining the presence of one or more features of interest or the presence of actions taken during a medical procedure). The systems and methods disclosed herein can be used in a variety of applications, including for medical image analysis and diagnosis. Background Art

[0002] In video processing and image analysis systems, it is often desirable to detect objects or features of interest. Features of interest can be people, places, or things. In some applications, for example, for systems and methods for medical image analysis, the location and classification of detected features of interest (e.g., such as abnormalities formed in or on human tissue) are important for the diagnosis of patients. However, existing computer-implemented systems and methods have many disadvantages, including the inability to accurately detect features of interest and / or identify characteristics associated with features of interest. In addition, existing systems and methods are inefficient and do not provide ways to intelligently analyze images, including with respect to the presence of image sequences or events.

[0003] Modern medical procedures require precise and accurate examination of the patient's body and organs. Endoscopy is a medical procedure designed to provide physicians with video images of the inside of the patient's body and organs for diagnosis. In the gastrointestinal tract of the human body, a probe with a camera can be introduced via the patient's mouth or anus to perform the procedure. During the endoscopic procedure, the physician manually navigates the probe through the gastrointestinal tract while watching the video on the display device in real time. Video can also be captured, stored and checked after the endoscopic procedure. As another option, capsule endoscopy is a process in which a capsule containing a small camera is swallowed to examine the patient's gastrointestinal tract. The image sequence taken by the capsule during its transportation is wirelessly transmitted to a receiving device and stored for physician inspection after the procedure is completed. The frame rate of the capsule device can vary (e.g., 2 to 6 frames per second), and a large number of images can be taken during the inspection process.

[0004] From the perspective of computer vision, the content captured from real-time video endoscopy or capsule inspection process is a temporally ordered sequence of images containing information about the patient (e.g., the internal mucosa of the gastrointestinal tract). Accurately and precisely analyzing the captured image data is essential for correctly examining the patient and identifying lesions, polyps or other features of interest. In addition, a large number of images are usually collected for each patient. One of the most important medical tasks that needs to be performed by a physician is to check this large group of images to make appropriate diagnosis, including about the presence or absence of features of interest, such as pathological areas in the imaging mucosa. However, manually browsing these images is both time-consuming and inefficient. As a result, this inspection process may cause the physician to make mistakes and / or make wrong diagnoses.

[0005] In order to improve diagnosis, reduce the time required for medical image examination, and reduce the possibility of error, the inventors have determined that it is desirable to have such a computer-implemented system and method that can intelligently process images and identify the presence of pathology or other features of interest within all images from video endoscopy or capsule examination procedures or other medical procedures. As an example, features of interest may also include actions taken on or in an image, anatomical locations or other locations of interest in an image, clinical indicator levels of an image, etc. To this end, trained neural networks, spatiotemporal image analysis, and other features and techniques are disclosed herein. As will be understood from this disclosure, the present invention and embodiments can be applied to a variety of image capture and analysis applications and are not limited to the examples presented herein. Summary of the invention

[0006] Embodiments of the present disclosure include systems, methods, and computer-readable media for processing images captured from imaging devices and performing intelligent image analysis (e.g., determining the presence of one or more features of interest). Systems and methods consistent with the present disclosure can provide benefits to existing systems and technologies, including by addressing one or more of the above-referenced shortcomings and / or other shortcomings of existing systems and technologies. Systems, methods, and computer-readable media consistent with some disclosed embodiments are provided for processing images from video endoscopy or capsule inspection processes or other medical processes (where images are temporally ordered). Exemplary embodiments include systems and methods for intelligently processing captured images using spatiotemporal information to accurately assess the likelihood of the presence of abnormalities, pathologies, or other features of interest within the image. As another example, the feature of interest can be a parameter or statistic associated with an endoscope or capsule process or other medical process. As an example, the feature of interest of an endoscopy process can be the clean-up withdrawal time or travel time of a probe or capsule through an organ. Features of interest in an image can also be determined based on the presence or absence of characteristics associated with the feature of interest. These and other embodiments, features, and implementations are described more fully herein. A feature of interest may be any feature in or associated with one or more images, particularly any feature in or associated with a scene or field of view represented in one or more images, which may be identified or detected by analyzing each image. A feature of interest may be, for example, an object, location, action, or condition (e.g., a clinical indicator level).

[0007] In some embodiments, the image captured by an imaging device (such as an endoscopic camera or a capsule camera) includes an image of the gastrointestinal tract or an organ. The image may come from a medical imaging device used during, for example, a gastroscopy, a colonoscopy, or an enteroscopy. For example, the feature of interest in the image may be an abnormality or other pathology. The abnormality or pathology may include a formation on a human tissue or a formation of a human tissue, a change in a human tissue from one type of cell to another type of cell, a lack of human tissue at a location where human tissue is expected, or a formation on a human tissue or a formation of a human tissue. The formation may include lesions, polyp lesions, or non-polyp lesions. Other examples of features of interest include anatomical or other locations, actions, clinical indicators (e.g., cleanliness), etc. Therefore, as will be understood from the present disclosure, exemplary embodiments may be utilized in a medical context in a manner that is not specific to any single disease but in a generally applicable manner.

[0008] According to one general aspect of the present disclosure, a computer-implemented system for processing images captured by an imaging device is provided. The computer-implemented system may include at least one processor configured to detect at least one feature of interest in an image captured by an imaging device. The at least one processor may be configured to: receive an ordered set of images from the captured images, the ordered set of images being ordered in time; analyze one or more subsets of the ordered set of images separately using a local spatiotemporal processing module, the local spatiotemporal processing module being configured to determine the presence of a characteristic associated with at least one feature of interest in each image of each image subset, and annotate the subset images with a feature vector based on the determined characteristic in each image of each image subset; process a feature vector set of the ordered set of images using a global spatiotemporal processing module, the global spatiotemporal processing module being configured to refine the determined characteristics associated with each image subset, wherein each feature vector of the feature vector set includes information associated with each determined characteristic of at least one feature of interest; and calculate a numerical value for each image using a time series analysis module, the numerical value representing the presence of at least one feature of interest and calculated using the refined features associated with each image subset and the spatiotemporal information. In addition, the at least one processor can be configured to generate a report about the at least one feature of interest using the numerical value associated with each image of each subset of the ordered set of images. The report can be generated after the endoscopy or other medical procedure is completed. The report can include information related to all features of interest identified in the processed images.

[0009] The at least one processor of the computer-implemented system may also be configured to determine a likelihood of a characteristic associated with at least one feature of interest in each image of the image subset. Additionally, the at least one processor may be configured to determine the likelihood of the characteristic in each image of the image subset by encoding each image of the image subset and aggregating spatiotemporal information of the determined characteristic using a recurrent neural network or a temporal convolutional network.

[0010] To refine the determined characteristics, a non-causal temporal convolutional network can be utilized. For example, at least one processor of the system can be configured to refine the likelihood of the characteristic in each image of the subset of images by applying a non-causal temporal convolutional network. The at least one processor can also be configured to refine the likelihood of the characteristic by applying one or more signal processing techniques including, for example, low pass filtering and / or Gaussian smoothing.

[0011] According to yet another aspect, at least one processor of the system may be configured to analyze the ordered set of images using a local spatiotemporal processing module to determine the presence of a feature by determining a vector of quality scores, wherein each quality score in the vector of quality scores corresponds to each image of a subset of images. In addition, at least one processor may be configured to process the ordered set of images using a global spatiotemporal processing module by refining the quality scores of each image of the image subset of one or more subsets of the ordered set of images using signal processing techniques. The at least one processor may also be configured to analyze the one or more subsets of the ordered set of images using a local spatiotemporal processing module to determine the presence of a feature by generating a pixel-by-pixel binary mask for each image of the image subset using a deep convolutional neural network. The at least one processor may also be configured to process the one or more subsets of the ordered set of images using a global spatiotemporal processing module by refining a binary mask for image segmentation using morphological operations that utilize prior information about the shape and distribution of the determined features.

[0012] As disclosed herein, various implementations may include one or more of the following features. The probability of a determination of a feature in each image of the image subset may include a floating value between 0 and 1. The quality score may be an ordinal number between 0 and R, where a score of 0 represents minimum quality and a score of R represents maximum quality. The numerical value may be associated with each image to determine a probability of identifying at least one feature of interest within the image. The output may be a first numerical value for an image in which at least one feature of interest is not detected. The output may be a second numerical value for an image in which at least one feature of interest is detected. The size or volume of the image subset may be configurable by a user of the system. The size or volume of the image subset may be dynamically determined based on the requested feature of interest. The size or volume of the image subset may be dynamically determined based on the determined characteristic. One or more image subsets may include shared images.

[0013] Another general aspect of the present disclosure relates to a computer-implemented system for performing spatiotemporal analysis of images captured with an imaging device. The computer-implemented system may include at least one processor configured to receive a video captured from an imaging device including a plurality of image frames. The at least one processor may also be configured to: access a temporally ordered set of images from the captured images; detect the occurrence of an event in the temporally ordered set of images using an event detector module, wherein a start time and an end time of the event are identified by a start image frame and an end image frame in the temporally ordered set of images; select a set of images from a temporally ordered set of images in the image set bounded by a start image frame and an end image frame using a frame selector module based on association scores and quality scores of the images, wherein the association scores of the selected images indicate the presence of at least one feature of interest; merge a subset of images from the selected images based on the matching presence of at least one feature of interest using an object descriptor module, wherein the subset of images is identified based on spatial and temporal correlation using spatiotemporal information; and segment the temporally ordered set of images at time intervals that satisfy the temporal correlation of a selected task.

[0014] According to the disclosed system, at least one processor may also be configured to use a local spatiotemporal processing module to determine spatiotemporal information of characteristics associated with at least one feature of interest of a subset of images of the video content, and to use a global spatiotemporal processing module to determine spatiotemporal information of all images of the video content. In addition, the at least one processor may be configured to segment the temporally ordered set of images by time intervals by identifying a subset of the temporally ordered set of images in the presence of at least one feature of interest. The at least one processor may also be configured to identify a subset of the temporally ordered set of images having the at least one feature of interest by adding a bookmark to an image in the temporally ordered set of images, wherein the bookmark image is part of the subset of the temporally ordered set of images. Additionally or alternatively, the at least one processor may be configured to identify a subset of the temporally ordered set of images having the presence of the at least one feature of interest by extracting a set of images from the subset of the temporally ordered set of images.

[0015] Various implementations may include one or more of the following features. The extracted image set may include characteristics associated with at least one feature of interest. For the at least one feature of interest, color may vary with a level of relevance of images of a subset of the temporally ordered image set. Color may vary with a level of relevance of images of a subset of the temporally ordered image set for characteristics associated with the at least one feature of interest.

[0016] Another general aspect includes a computer-implemented system for performing multiple tasks on a collection of images. The computer-implemented system may include at least one processor configured to receive a video captured from an imaging device including frames of a collection of images. The at least one processor may also be configured to: receive a plurality of tasks, wherein at least one of the plurality of tasks is associated with a request to identify at least one feature of interest in the collection of images; analyze a subset of images of the collection of images using a local spatiotemporal processing module to identify the presence of a characteristic associated with at least one feature of interest; and iteratively execute a time series analysis module for each of the plurality of tasks to associate a numerical score for each task with each image of the collection of images.

[0017] Consistent with the present disclosure, a system of one or more computers may be configured to perform operations or actions by installing software, firmware, hardware, or a combination thereof for the system, which in operation causes the system to perform these operations or actions. One or more computer programs may be configured to perform operations or actions by including instructions, which when executed by a data processing device (e.g., one or more processors) causes the device to perform these operations or actions.

[0018] Any suitable combination of software, firmware, and hardware may be used to implement systems and methods consistent with the present disclosure. Implementations of the present disclosure may include programs or instructions that are machine constructed and / or programmed specifically for performing functions associated with the disclosed operations or actions. In addition, a non-transitory computer-readable storage medium storing program instructions may be used, which may be executed by at least one processor to perform the steps and / or methods described herein.

[0019] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The following drawings, which comprise a part of this specification, illustrate several embodiments of the present disclosure and, together with the description, are used to explain the principles and features of the disclosed embodiments. In the drawings:

[0021] Figure 1A is a block diagram of an exemplary smart detector system consistent with embodiments of the present disclosure.

[0022] Figure 1B is a schematic representation of an exemplary computer-implemented system for processing real-time video consistent with embodiments of the present disclosure.

[0023] Figure 2 shows that it can be implemented with Figure 1A or Figure 1Bor an exemplary computing device used in conjunction with the exemplary system of other embodiments of the present disclosure.

[0024] Figure 3A is a block diagram of an exemplary local spatiotemporal processing module consistent with embodiments of the present disclosure.

[0025] Figure 3B is a block diagram of an exemplary global spatiotemporal processing module consistent with embodiments of the present disclosure.

[0026] Figure 3C is a block diagram of an exemplary time series analysis module consistent with embodiments of the present disclosure.

[0027] Figure 4A It is consistent with the embodiment of the present disclosure. Figure 3A Flowchart of exemplary training of the encoder component of the local spatiotemporal processing module.

[0028] Figure 4B It is consistent with the embodiment of the present disclosure. Figure 3A Flowchart of an exemplary training of a neural network component of a local spatiotemporal processing module.

[0029] Figure 4C It is consistent with the embodiment of the present disclosure. Figure 3A Flowchart of exemplary training of the mass network and segmentation network components of the local spatiotemporal processing module.

[0030] Figure 4D It is consistent with the embodiment of the present disclosure. Figure 3B Flowchart of an exemplary training of a global spatiotemporal processing module.

[0031] Figure 5A and Figure 5B is a schematic representation of a circuit constructed with components of an exemplary smart detector system consistent with embodiments of the present disclosure.

[0032] Fig. 6A and Figure 6B Different circuit configurations are shown for performing multiple tasks using an exemplary smart detector system consistent with embodiments of the present disclosure.

[0033] Figure 6C and Fig.6D An exemplary circuit setup for performing multiple tasks with aggregate output using an exemplary smart detector system consistent with embodiments of the present disclosure is shown.

[0034] Fig. 6E An exemplary panel consistent with an embodiment of the present disclosure is shown having output summaries of multiple tasks generated using an example smart detector system.

[0035] Figure 7 is a flow chart depicting the operation of an example method for analyzing images and detecting features of interest, consistent with embodiments of the present disclosure.

[0036] Figure 8 is a flow chart depicting the operation of an example method for spatiotemporal image analysis consistent with embodiments of the present disclosure.

[0037] Fig. 9 is a flow chart depicting the operation of an exemplary method for processing multiple tasks on a set of input images consistent with embodiments of the present disclosure.

[0038] Fig.10 An exemplary panel consistent with an embodiment of the present disclosure is shown having a summary of output generated using an exemplary smart detector system. DETAILED DESCRIPTION

[0039] Exemplary embodiments are described below with reference to the accompanying drawings. These drawings are not necessarily drawn to scale. Although examples and features of the disclosed principles are described herein, modifications, adaptations, and other implementations thereof are also feasible without departing from the spirit and scope of the disclosed embodiments. In addition, the terms "include," "have," "include," and other similar forms have equivalent meanings and are open ended, because a term or terms following any one or more of these terms are not intended to be exhaustive or limited to the terms listed. It should also be noted that as used herein and in the appended claims, the singular forms of "a," "an," and "the" include plural references unless the context clearly states otherwise.

[0040] In the following description, various working examples are provided for illustrative purposes. However, it should be understood that the present disclosure may be practiced without one or more of these details.

[0041] Throughout this disclosure, references to "disclosed embodiments" refer to examples of the inventive ideas, concepts, and / or manifestations described herein. Many related and unrelated embodiments are described throughout this disclosure. The fact that some "disclosed embodiments" are described as exhibiting a feature or characteristic does not mean that other disclosed embodiments must share that feature or characteristic.

[0042] The embodiments described herein include non-transient computer-readable media, which contain instructions that cause the at least one processor to perform a method or set of operations when executed by at least one processor. Non-transient computer-readable media can be any medium that can store data in any memory in a manner that can be read by any computing device with a processor to perform a method or any other instruction stored in the memory. Non-transient computer-readable media can be implemented as software, firmware, hardware, or any combination thereof. Software can preferably be implemented as an application tangibly embodied on a program storage unit or a computer-readable medium, which is composed of a combination of some or some devices and / or devices. The application can be uploaded to and executed by a machine including any suitable architecture. Preferably, the machine can be implemented on a computer platform with hardware such as one or more central processing units (CPUs), memory, and input / output interfaces. The computer platform can also include an operating system and microinstruction code. The various processes and functions described in this disclosure can be a part of a microinstruction code or a part of an application program or any combination thereof, which can be executed by a CPU, whether or not such a computer or processor is explicitly shown. In addition, various other peripheral units can be connected to the computer platform, such as additional data storage units and printing units. Furthermore, a non-transitory computer readable medium may be any computer readable medium except a transitory propagating signal.

[0043] Memory can include any mechanism for storing electronic data or instructions, including random access memory (RAM), read-only memory (ROM), hard disk, optical disk, magnetic media, flash memory, other permanently fixed volatile or non-volatile memory. Memory can include one or more independent storage devices that are collocated or allocated, which can store data structures, instructions or any other data. Memory can also include a memory portion containing instructions for execution by a processor. Memory can also be used as a working storage device or temporary storage for a processor.

[0044] Some embodiments may involve at least one processor. A processor may be any physical device or group of devices having circuits that perform logical operations on one or more inputs. For example, at least one processor may include one or more integrated circuits (ICs), including application specific integrated circuits (ASICs), microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a server, a virtual server, or other circuits suitable for executing instructions or performing logical operations. The instructions executed by at least one processor may be preloaded, for example, into a memory integrated with or embedded in a controller, or may be stored in a separate memory.

[0045] In some embodiments, the at least one processor may include more than one processor. Each processor may have a similar configuration, or the processors may have different configurations that are electrically connected or disconnected from each other. For example, the processors may be separate circuits or integrated into a single circuit. When more than one processor is used, the processors may be configured to operate independently or collaboratively. The processors may be electrically coupled, magnetically coupled, optically coupled, acoustically coupled, mechanically coupled, or coupled by other means that allow them to interact.

[0046] Embodiments consistent with the present disclosure may involve a network. The network may constitute any type of physical or wireless computer networking device for exchanging data. For example, the network may be the Internet, a private data network, a virtual private network using a public network, a Wi-Fi network, a LAN or WAN network, and / or other suitable connections that can enable information exchange between various components of the system. In some embodiments, the network may include one or more physical links for exchanging data, such as Ethernet, coaxial cable, twisted pair, optical fiber, or any other suitable physical medium for exchanging data. The network may also include one or more networks, such as a private network, a public switched telephone network (PSTN), the Internet, and / or a wireless cellular network. The network may be a secure network or an unsecured network. In other embodiments, one or more components of the system may communicate directly via a dedicated communication network. Any suitable technology may be used to communicate directly, including, for example, BLUETOOTH. TM 、BLUETOOTH LE TM (BLE), Wi-Fi, Near Field Communication (NFC), or other suitable communication methods that provide a medium for exchanging data and / or information between independent entities.

[0047] In some embodiments, a machine learning network or algorithm may be trained using training examples, such as in the case described below. Some non-limiting examples of such machine learning algorithms may include classification algorithms, video classification algorithms, data regression algorithms, image segmentation algorithms, temporal video segmentation algorithms, visual detection algorithms (such as object detectors, face detectors, human detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (such as face recognition, human recognition, object recognition, etc.), speech recognition algorithms, action recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbor algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recursive neural network algorithms, linear machine learning models, nonlinear machine learning models, integrated algorithms, etc. For example, a trained machine learning network or algorithm may include an inference model, such as a prediction model, a classification model, a regression model, a clustering model, a segmentation model, an artificial neural network (such as a deep neural network, a convolutional neural network, a recursive neural network, etc.), a random forest, a support vector machine, etc. In some examples, a training example may include an example input together with an expected output corresponding to the example input. In addition, in some examples, a trained machine learning algorithm using a training example may generate a trained machine learning algorithm, and the trained machine learning algorithm may be used to estimate the output of an input not included in the training example. Training may be supervised or unsupervised, or a combination thereof. In some examples, engineers, scientists, processes, and machines that train machine learning algorithms may further use verification examples and / or test examples. For example, verification examples and / or test examples may include example inputs and expected outputs corresponding to the example inputs, and the trained machine learning algorithm and / or the intermediately trained machine learning algorithm may be used to estimate the output of the example inputs of the verification examples and / or test examples, and the estimated output may be compared with the corresponding expected output, and the trained machine learning algorithm and / or the intermediately trained machine learning algorithm may be evaluated based on the results of the comparison. In some examples, the machine learning algorithm may have parameters and hyperparameters, wherein the hyperparameters are manually set by a person or automatically set by a process external to the machine learning algorithm (such as a hyperparameter search algorithm), and the parameters of the machine learning algorithm are set by the machine learning algorithm according to the training examples. In some implementations, hyperparameters are set according to the training examples and the verification examples, and parameters are set according to the training examples and the selected hyperparameters. The machine learning network or algorithm may be further retrained based on any output.

[0048] Certain embodiments disclosed herein may include a computer-implemented system for performing an operation or method comprising a series of steps. The computer-implemented system and method may be implemented by one or more computing devices, which may include one or more processors configured to process real-time video as described herein. The computing device may be one or more computers or any other device capable of processing data. Such a computing device may include a display such as an LCD display, an augmented reality (AR) or a virtual reality (VR) display. However, the computing device may also be implemented in a computing system including a back-end component (e.g., as a data server), or a computing system including a middleware component (e.g., an application server), or a computing system including a front-end component (e.g., a user device with a graphical user interface or a web browser, through which a user can interact with the implementation of the system and technology described herein), or any combination of such back-end, middleware or front-end components. The components of the system and / or computing device may be interconnected by digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet. The computing device may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0049] Figure 1A is a block diagram of an exemplary smart detector system 100 consistent with an embodiment of the present disclosure. As further disclosed herein, the smart detector system 100 can be a computer-implemented system and includes one or more convolutional neural networks (CNNs) for processing images from a medical procedure to identify requested features of interest in the image. One or more features of interest can be a pathology or list of pathologies that a physician is looking for in an image (e.g., for diagnosing a patient). As a further example, features of interest can also include actions to be taken on or in an image, anatomical locations or other locations of interest in an image, clinical indicator levels of an image, and the like. These and other examples are within the scope of the present disclosure. As an example, an action can include an action taken by a physician during a medical procedure or as part of a subsequent procedure, including an action or process identified by the system 100 as a result of a spatiotemporal examination of an image from a medical procedure. For example, an action can include a recommended action or procedure according to a medical guideline, such as performing a biopsy, removing a lesion, or exploring / analyzing the surface / mucosa of an organ. An action or process can be identified based on images captured and processed by the smart detector system 100.

[0050] The smart detector system 100 may receive as input a collection of time-ordered images of a medical procedure (e.g., an endoscopy or colonoscopy procedure). The smart detector system 100 may output a report or information including one or more numerical values ​​(e.g., scores) for each image. The numerical value may relate to a medical category such as a specific pathology and provide information about the probability of the medical category existing within the image frame. The image processed by the smart detector system 100 may be an image captured from a medical procedure stored in a database or memory device for subsequent retrieval and processing by the smart detector system 100. In some embodiments, the output generated from the processed image provided by the smart detector system 100 may include, for example, a report with one or more numerical scores assigned to the image and subsequent steps recommended according to one or more medical guidelines. The report may be generated after the endoscopy or other medical procedure is completed. The report may include information related to all features of interest identified in the processed image. Further, in some embodiments, the output provided by the smart detector system 100 may include one or more recommended actions performed by a physician in view of one or more features of interest identified in the image from the medical procedure (e.g., performing a biopsy, removing a lesion, exploring / analyzing the surface / mucosa of an organ, etc.). During a medical procedure, the smart detector system 100 can receive video or image frames directly from the medical imaging device, process the video or image frames, and provide feedback to the operator during the procedure or just after the medical procedure (i.e., a short time interval ranging from no time to several minutes) regarding the actions performed by the operator, as well as a final report containing multiple measured variables, clinical indicators and details about what was observed during the medical procedure and / or at which anatomical location and / or how the operator behaved. The actions performed may include recommended actions or procedures according to medical guidelines, such as performing a biopsy, removing a lesion, or exploring / analyzing the surface / mucosa of an organ. In some embodiments, the recommended action can be part of a set of recommended actions based on the medical guidelines. In the following description Figure 1B A detailed description of an exemplary computer system implementing the smart detector system 100 for real-time processing is presented in .

[0051] As disclosed herein, the smart detector system 100 can generate a report after completing a medical procedure that includes information about the processing of the captured video based on the local spatiotemporal processing module 110, the global spatiotemporal processing module 120, and the time series analysis module 130. The report can include information related to features of interest identified during the medical procedure, as well as other information such as one or more numerical values ​​or scores for each image. As explained, the one or more numerical values ​​can relate to a medical category such as a specific pathology and provide information about the probability of the medical category being present within the image frame. Further details about the operation of the smart detector system 100 and the local spatiotemporal processing module 110, the global spatiotemporal processing module 120, and the time series analysis module 130 are provided below with reference to the accompanying drawings.

[0052] In some embodiments, the report generated by the system 100 may include one or more additional recommended actions based on the processing of stored images from the medical procedure or the real-time processing of images from the medical procedure. The one or more additional recommended actions include actions or procedures that may be taken during the medical procedure and actions or procedures taken after the medical procedure. The one or more additional recommended actions may be part of a set of recommended actions based on medical guidelines. In addition, as described above, the system 100 can process the video in real time to provide the operator with concurrent feedback on what is occurring or identified in the video and during the medical procedure.

[0053] The output generated by the smart detector system 100 may include a panel display or similar report (see, for example, Fig.10 ). The panel can provide a summary report of a medical procedure (e.g., an endoscopy or colonoscopy). The panel can provide a quality score and / or other information for the procedure. The score and / or other information can summarize the inspection behavior of the care professional and provide information for identified features of interest (e.g., the number of polyps identified). In some embodiments, the information generated by the system 100 is provided and displayed as an overlay on a video of the medical procedure, and thus the operator can view the information as part of enhanced video feedback during or just after the completion of the medical procedure. The information can be provided with some delay or without delay.

[0054] The smart detector system 100 can also generate reports in the form of electronic files, a set of data, or data transmission. As an example, the output generated by the system 100 can follow a standardized format and / or be integrated into records such as electronic health records (EHR). The output of the system 100 can also comply with regulations such as HIPAA for interoperability and privacy. In some embodiments, the output can be integrated into other reports. For example, the output of the smart detector system 100 can be integrated into the patient's electronic medical or health record. The smart detector system 100 may include an API to facilitate such integration and / or provide output in the form of a standardized data set or template. The standardized template may include a predefined table or chart that can be filled with data values ​​generated by the smart detector system 100 by processing the input video or image frames from the medical process. In some embodiments, the report can be generated by the system 100 in a machine-readable format (e.g., an XML file) to support its transmission or storage and integration with other systems and applications. In some embodiments, the report can be provided in other formats such as Word, Excel, HTML, or PDF file formats. In some embodiments, the smart detector system 100 can be accessed through a network (e.g., see Figure 1A The smart detector system 100 can also transmit the output data or formatted report (e.g., as a JSON document) to a server or database by making an API call and sending it.

[0055] Fig.10An exemplary panel 1000 having an output summary 1090 generated using an intelligent detector system (such as the intelligent detector system 100) consistent with an embodiment of the present disclosure is shown. Using the modules of the intelligent detector system 100, an output summary 1090 can be generated for the process. The output summary 1090 can provide quality scores and / or other information such as process time, extraction time, and clean-up extraction time. Other examples of information that can be part of the summary 1090 include the time spent performing specific actions (such as one or more recommended actions discussed above) and the time spent in different anatomical locations. Information related to features of interest (such as polyps) can also be provided. For example, the time series analysis module 130 can generate a summary of the number of polyps identified based on features observed in the image frames. The time series analysis module 130 can aggregate information generated by processing images of the input video using the local and global spatiotemporal processing modules 110 and 120. The summary panel 1000 can also include a visual description of the features of interest identified by the intelligent detector system 100. For example, selected video frames of the procedure may be enhanced with indicia such as green bounding boxes regarding the location of each identified feature of interest, as shown in frames 1092, 1094, and 1096. These frames may relate to different examined portions of the colon, such as the ileocaecal valve, foramen, and triradiate folds, which themselves may be features of interest requested by a user of the smart detector system 100. Although Fig.10 The example shows information of a procedure displayed as part of a single panel, but multiple panels may also be generated with output summaries for each portion of the colon or other organ examined as part of the medical procedure. In some embodiments, a combined score or value is generated based on input received as a plurality of vectors (e.g., image score vectors) generated by local and global spatiotemporal processing modules 110 and 120.

[0056] As disclosed above, the features of interest may relate to a medical category or pathology. The smart detector system 100 may be implemented to handle requests to detect one or more medical categories (i.e., one or more features of interest) in an image. In the case of multiple features of interest, one instance of the components of the smart detector system 100 may be implemented for each medical category or feature of interest. As will be understood from this disclosure, an instance of the smart detector system 100 may be implemented in any combination of hardware, firmware, and software depending on speed or insertion requirements, the volume of images to be processed, and other requirements of the system.

[0057] In some embodiments, a single instance of the smart detector system 100 may output multiple numerical values ​​for each image, one for each medical category. In one exemplary embodiment, the pathology detected by the smart detection system 100 may include detecting polyps in the colon mucosa. Further, as an example, the smart detector system 100 may output a numerical value (e.g., 0) for all images in the input images in which the smart detector system 100 did not detect a polyp, and may output another numerical value (e.g., 1) for all images in the input images in which the smart detector system 100 detected at least one polyp. In some embodiments, the numerical values ​​may be arranged relative to a range or scale and / or indicate the probability of the presence of a polyp or other feature of interest.

[0058] The source of the input image can vary depending on the needs of the imaging device, storage device, and / or application. For example, the smart detector system 100 can be configured to directly process video feedback from a video endoscope device and receive temporally ordered input images that are subsequently processed by the system, consistent with the embodiments disclosed herein. As another example, the smart detector system 100 can be configured to receive input images from a database or memory device, the stored images being temporally ordered and previously captured using an imaging device (such as a camera of an endoscope device or a camera of a capsule device). The images received by the smart detector system 100 can be processed and analyzed to identify one or more features of interest, such as one or more types of polyps or lesions.

[0059] Can be implemented in various environments and for various applications Figure 1AAn exemplary system of. For example, the captured input images may be stored in a local database or memory device, or they may be accessed and received by the smart detector system 100 from a remote storage location (such as cloud storage) via a network. The smart detector system 100 may also be configured to process streaming video feedback from a current medical procedure, and the processing input images are collected from the feedback (e.g., via preprocessing and caching). In addition, the operation of the smart detector system 100 may be programmed or triggered to start under one or more conditions. For example, the smart detector system 100 may be configured to directly analyze the input image when receiving an input image (e.g., via video feedback or a set of stored input images from a memory) or when receiving a command from a user. The output of the smart detector system 100 may also be configured on demand. For example, as previously described, the smart detector system 100 may analyze the input image of one or more features of interest, and generate a report indicating the presence of one or more features of interest in the processed image. The report may take the form of an electronic file, a graphical display, and / or an electronic transmission of data. It should be understood that other output and report formats are within the scope of the present disclosure. In some embodiments, reports in different formats may be preconfigured and used as templates to generate reports by populating the templates with values ​​generated by the smart detector system 100. In some embodiments, the reports are formatted for integration into other reporting systems, such as electronic medical records (EMRs). The report format may be a machine-readable format, such as XML or Excel, for integration with other reporting systems.

[0060] As an example, the smart detector system 100 can process recorded video or images and provide a fully automated report and / or other output that details the features of interest observed in the processed images. The smart detector system 100 can use artificial intelligence or machine learning components to efficiently and accurately process input images and make decisions about the presence of features of interest based on image analysis and / or spatiotemporal information. In addition, for each feature of interest requested or under investigation, the smart detector system 100 can estimate its presence within the image and provide information to the report or other output indicating the likelihood and other details of the presence of the feature, such as the relative time from the beginning of the process or image sequence in which the feature of interest appeared, the estimated anatomical location, the duration, the most important images, the location within these images, and / or the number of occurrences.

[0061] In one embodiment, the smart detector system 100 can be configured to automatically determine the presence of gastrointestinal pathology without the help of a physician. As discussed above, the input image can be captured and received in different ways and using different types of imaging devices. For example, a video endoscope device or a capsule device or other medical device or other imaging device can record and provide an input image. The input image can be part of a live video feed, or can be part of a stored image collection received from a local or remote storage location (e.g., a local database or cloud storage). The smart detector system 100 can be operated as part of a process or service in a clinic or hospital, or can be provided as an online or cloud service for end users to enable self-diagnosis or remote testing.

[0062] For example, to begin the inspection process, the user can ingest a capsule device or pill box. The capsule device may include an imaging device and wirelessly transmit images of the user's gastrointestinal tract to a smartphone, tablet, laptop, computer, or other device (e.g., user device 170) during the process. The captured image may then be uploaded via a network connection to a database, cloud storage, or other storage device (e.g., image source 150). The smart detector system 100 may receive an input image from an image source and analyze the image for one or more requested features of interest (e.g., polyps or lesions). The final report may then be provided electronically to the user and / or their physician as an output. The report may include a score or probability index for each observed feature of interest and / or other relevant information or medical advice. In addition, the smart detector system 100 may detect pathophysiological features associated with the feature of interest and indicators of the feature of interest, and score those features determined to be present. Examples of such features include bleeding, inflammation, ulcers, tumor tissue, and the like. In addition, in response to the detected feature of interest, the report may include information or advice based on medical guidelines, such as advice to consult a physician and / or perform additional diagnostic tests. One or more actions (e.g., performing a biopsy, removing a lesion, exploring / analyzing the surface / mucosa of an organ, etc.) may also be recommended to the physician based on the analysis of the images by the smart detector system 100, either in real time during the medical procedure or after the medical procedure is completed.

[0063] As another example, the smart detector system 100 can help a physician or specialist analyze video content recorded during a medical procedure or examination. The captured image can be a portion of video content recorded during, for example, a gastroscopy, colonoscopy, or enteroscopy procedure. Based on the analysis performed by the smart detector system 100, the full video recording can be displayed to the physician or specialist along with a colored timeline bar, where different colors correspond to different features of interest and / or scores of identified features of interest.

[0064] As a further example, a physician, expert, or other individual may use the smart detector system 100 to create a summary of a video record or image collection by focusing on images having a desired feature of interest and discarding irrelevant image frames. The smart detector system 100 may be configured to allow the physician or user to tune or select features of interest for detection and the duration of each summary based on total duration and / or other parameters, such as preset pre- and post-times before and after a sequence of frames having a selected feature of interest. The smart detector system 100 may also be configured to combine all or the most relevant frames according to a requested feature of interest.

[0065] like Figure 1A As shown, the smart detector system 100 may include a local spatiotemporal processing module 110, a global spatiotemporal processing module 120, a time series analysis module 130, and a task manager 140. These components may be implemented by any suitable combination of hardware, software, and / or firmware. In addition, the number and arrangement of these components may be modified, and it should be understood that the above are provided for illustrative purposes. Figure 1A The following are exemplary embodiments of the present invention and do not limit the scope of the present invention and its embodiments. Figure 1B and FIG. 3A to FIG. 3C Other example features and details associated with these components.

[0066] Reference again Figure 1A In an exemplary embodiment of the present invention, the local spatiotemporal processing module 110 can be configured to provide a local perspective by processing a subset of images of an input video or a set of input images. The local spatiotemporal processing module 110 can select a subset of images and process the images to generate a score based on the presence of a feature associated with one or more features of interest. For example, assume that an endoscopy input video V includes a set of T image frames. The feature can define a feature of interest requested by a user of the intelligent detector system 100. For example, the feature can include physical and / or biological aspects, such as the size, orientation, color, shape, etc. of the feature of interest. The feature can also include metadata, such as data identifying a portion of a video or a time period in a video. For example, the feature of a colonoscopy procedure video can identify one or more parts of the colon, such as the ascending colon, the transverse colon, or the descending colon. In another example, the feature can relate to one or more parts of an endoscopy procedure video, such as the amount of motion in the image, the presence of an instrument, or the duration of a motion reduction segment. The feature defining the content can also indicate the behavior of a physician, clinician, or other individual performing a medical procedure. For example, the longest portion of the video is paused without moving or exploring the surface of an organ for the longest time. In some embodiments, the feature can be a feature of interest. For example, the feature or characteristic of interest for a colonoscopy procedure video may be a portion of the colon, such as the ascending colon, the transverse colon, or the descending colon.

[0067] The local spatiotemporal processing module 110 may be configured to process the entire input video in blocks by iterating over successive batches or subsets of N image frames. The local spatiotemporal processing module 110 may also be configured to provide an output including a vector or quality score representing the determined characteristics of the feature of interest in each image frame. In some embodiments, the local spatiotemporal processing module 110 may output a quality value and a segmentation map associated with each image frame. Figure 3A The embodiments of the present invention provide other exemplary details related to the local spatiotemporal processing module 110.

[0068] The image subsets processed by the local spatiotemporal processing module 110 may include shared or superimposed images. In addition, the size or arrangement of the image subsets may be defined or controlled based on one or more factors. For example, the size or volume of the image subsets may be configured by a physician or other user of the system. As another example, the local spatiotemporal processing module 110 may be configured such that the size or volume of the image subsets is dynamically determined based on the requested features of interest. Additionally or alternatively, the size of the image subsets may be dynamically determined based on determined characteristics associated with the requested one or more features of interest.

[0069] The global spatiotemporal processing module 120 can be configured to provide a global perspective by processing all subsets of the images analyzed by the local spatiotemporal processing module 110. For example, the global spatiotemporal processing module 120 can process the entire input video or input image set by processing all outputs of the local spatiotemporal processing modules 110 simultaneously or together. In addition, the global spatiotemporal processing module 120 can be configured to provide an output including a numerical score for each image frame by processing a vector of determined characteristics associated with one or more features of interest. In some embodiments, the global spatiotemporal processing module 120 can process images and vectors and output a refined quality score and segmentation map for each image. Figure 3B Further exemplary details regarding the global spatiotemporal processing module 120 are provided in the following embodiments.

[0070] The time series analysis module 130 uses information about the image determined by the local spatiotemporal processing module 110 and refined by the global spatiotemporal processing module 120 to output a numerical score to indicate the presence of one or more features of interest requested by a user of the smart detector system 100. For example, the time series analysis module 130 can be configured to perform a time series analysis on an input video or image using spatiotemporal information of characteristics associated with the features of interest determined by the local spatiotemporal processing module 110. Figure 3C Further exemplary details regarding the time series analysis module 130 are provided in the following embodiments.

[0071] The task manager 140 can help manage various tasks requested by users of the smart detector system 100. Tasks can involve requested or required features of interest and / or characteristics of features of interest. One or more features of interest and characteristics can be part of each task for processing by the smart detector system 100. The task manager 140 can help manage tasks for detecting multiple features of interest in a set of input images. The task manager 140 can determine the number of instances of components of the smart detector system 100 (e.g., the local spatiotemporal processing module 110, the global spatiotemporal processing module 120, and the time series analysis module 130). Referring to FIGS. 6 and 7 below, Figure 7 A description of the present invention is provided to provide further exemplary details of a manner of processing multiple task requests to detect features of interest.

[0072] The smart detector system 100 may receive an input video or image set from an image source 150 via a network 160 for processing. In some embodiments, the smart detector system 100 may receive an input video directly from another system, such as a medical instrument or system used to capture video when performing a medical procedure, colonoscopy. After processing the image, a report of the detected features of interest may be shared via the network 160. As disclosed herein, the report may be transmitted electronically and take different forms, such as electronic files, displays, and data. In some embodiments, the report is sent as a file to a user device 170 and / or displayed at the user device 170. The network 160 may take various forms depending on system requirements and circumstances. For example, the network 160 may include or utilize any combination of the Internet, a wired wide area network (WAN), a wired local area network (LAN), a wireless WAN (e.g., WiMAX), a wireless LAN (e.g., IEEE 802.11, etc.), a mesh network, a mobile / cellular network, an enterprise or private data network, a storage area network, a virtual private network using a public network, and / or other types of network communications. In some embodiments, network 160 may include a local (e.g., LAN) network, while in other embodiments, network 160 may include a virtualized, remote, and / or cloud network (e.g., AWS TM 、Azure TM , IBM Cloud TM In addition, in some embodiments, network 160 may be a hybrid local and virtualized, remote and / or cloud network, including components of one or more types of network architectures.

[0073] The user device 170 can send a request to the smart detector system 100 and receive output (e.g., a report or data) related to one or more features of interest in the input video or image from the smart detector system 100. The user device 170 can control the input video or image or directly provide the input video or image to the smart detector system 100 for processing, which includes downloading and / or storing a link to a storage location (e.g., image source 150) through instructions, commands, video or image collection files. The user device 170 may include a smartphone, a laptop, a tablet, a computer, and / or other computing devices. The user device 170 may also include an imaging device (e.g., a video or digital camera) to capture video or images for processing. For example, in the case of a capsule inspection process, the user device 170 includes a pill box or the like, which is ingested by the user and causes the input video or image to be captured and streamed directly to the smart detector system 100 or stored in the image source 150, and then downloaded and received by the system 100 via the network 160. The result of the image processing is then provided to the user device 170 via the network 160 as an output from the smart detector system 100.

[0074] The physician device 180 may also be used to send requests to the smart detector system 100 and receive output (e.g., reports or data) from the smart detector system 100 related to one or more features of interest in the input video or image. Similar to the user device 170, the physician device 180 may control the input video or image or directly provide the input video or image to the smart detector system 100 for processing, which includes downloading and / or storing links to a storage location (e.g., image source 150) through instructions, commands, video or image collection files. The physician device 180 may include a smartphone, a laptop, a tablet, a computer, and / or other computing devices. The physician device 180 may also include an imaging device (e.g., a video or digital camera) to capture video or images for processing. In the case of video endoscopy, for example, the physician device 180 may include a colonoscopy probe or an imaging device similar to that used to capture images during a patient examination. The captured video may be transmitted to the smart detector system 100 as an input video stream or stored in the image source 150 and subsequently downloaded and received by the system 100 via the network 160. In some embodiments, the physician device 180 may receive a notification for further inspection of the image frame having the feature of interest. The results of the image processing are then provided as output (e.g., an electronic report or data in the form of a file or digital display) from the smart detector system 100 to the user device 170 via the network 160.

[0075] Image source 150 may include a storage location or other source for inputting video or images to smart detector system 100. Image source 150 may include any suitable combination of hardware, software, and firmware. For example, image source 150 may include any combination of computing devices, servers, databases, memory devices, network communication hardware, and / or other devices. By way of example, image source 150 may include a database, memory, or storage (e.g., Figure 2 The image source 150 storage may include file storage and / or use of a CPU (e.g., Figure 2 As another example, the image source 150 may also include a cloud storage accessible via the network 160, such as AMAZON TM S3, Azure TM Memory, GOOGLE TM Cloud storage.

[0076] exist Figure 1A In the exemplary system of FIG. 1 , the image source 150, the user device 170, and the physician device 180 may be local or remote from each other and may communicate with each other via wired or wireless communications (including via network communications). Depending on the application and needs of the system implementation, the devices may also be local or remote from the smart detector network 100. In addition, although Figure 1A The image source 150, user device 170, and physician device 180 are shown as being separate from the smart detector system 100, but one or more of these devices may be local or provided as part of the system 100. Also, some or part of the network 160 may be local or part of the system 100. Furthermore, it should be understood that providing Figure 1A The number and arrangement of components and devices in the drawings are for illustrative purposes and are not intended to limit the invention or the disclosed embodiments thereof.

[0077] Although embodiments of the present disclosure are described herein with reference to medical image analysis and endoscopy, it should be understood that these embodiments can be applied to other medical image procedures, such as gastroscopy, colonoscopy, and enteroscopy. In addition, embodiments of the present disclosure can be implemented for other image capture and analysis environments and systems, such as for or including LIDAR, surveillance, automatic navigation, and other imaging systems.

[0078] According to one aspect of the present disclosure, a computer-implemented system is provided for intelligently processing an input video or image set and determining the presence of a feature of interest and characteristics associated therewith. As further disclosed herein, a system (e.g., smart detector system 100) may include at least one memory (e.g., ROM, RAM, local memory, network memory, etc.) configured to store instructions and at least one processor (e.g., processor 230) configured to execute instructions (e.g., see FIGS. 1 and 2). Figure 2 ). Using the at least one processor, the system can process an input video or image set captured by a medical imaging system, such as those used during an endoscopy, gastroscopy, colonoscopy, or enteroscopy procedure. Additionally or alternatively, the image frames can include medical images, such as images of gastrointestinal organs or other organs or regions of human tissue.

[0079] As used herein, the term "image frame" or "image" refers to any digital representation of a scene or field of view captured by an imaging device. The digital representation can be encoded in any appropriate format, such as a Joint Photographic Experts Group (JPEG) format, a Graphics Interchange Format (GIF), a bitmap format, a Scalable Vector Graphics (SVG) format, an Encapsulated PostScript (EPS) format, etc. Similarly, the term "video" refers to any digital representation of a scene or region of interest including multiple images in a sequence. The digital representation of a video can be encoded in any appropriate format, such as a Moving Picture Experts Group (MPEG) format, a Flash Video format, an Audio Video Interleaved (AVI) format, etc. In some embodiments, the image sequence for the input video can be paired with audio. As will be understood from the present disclosure, embodiments of the present invention are not limited to processing input videos having sequential or temporally ordered image frames, but can also process streaming or stored image collections captured in a sequential or temporally ordered manner. Therefore, the terms "input video" and "image collection" should be considered interchangeable and do not limit the scope of the present disclosure.

[0080] As disclosed herein, an image frame or image may include a representation of a feature of interest (i.e., an abnormality or other object of interest). For example, a feature of interest may include an abnormality on or in human tissue. In other embodiments of non-medical procedures, a feature of interest may include an object, such as a vehicle, a person, or other entity.

[0081] According to the present disclosure, an "abnormality" may include formations on human tissue or formation of human tissue, changes in human tissue from one type of cell to another type of cell, and / or, lack of human tissue at a location where human tissue is expected. For example, a tumor or other tissue growth may include an abnormality because there are more cells than expected. Similarly, a bruise or other change in cell type may include an abnormality because blood cells are present in locations other than the expected location (i.e., outside of the capillaries). Similarly, a depression in human tissue may include an abnormality because there are no cells at the expected location, resulting in a depression.

[0082] In some embodiments, the abnormality may include a lesion. The lesion may include a lesion of the gastrointestinal mucosa. Lesions may be classified histologically (e.g., according to Narrow Band Imaging International Colorectal Endoscopy (NICE) or Vienna classification), morphologically (e.g., according to Paris classification), and / or structurally (e.g., serrated or non-serrated). The Paris classification includes polyp lesions and non-polyp lesions. Polyp lesions may include prominent stalked lesions or prominent sessile lesions. Non-polyp lesions may include superficial protrusions, flats, superficial depressions, or pit lesions. With respect to detecting abnormalities as features of interest, serrated lesions may include sessile serrated adenomas (SSA); traditional serrated adenomas (TSA); hyperplastic polyps (HP); fibroblastic polyps (FP); or mixed polyps (MP). According to the NICE classification system, abnormalities are divided into the following three types: (Type 1) sessile serrated polyps or hyperplastic polyps; (Type 2) conventional adenomas; and (Type 3) cancers with deep submucosal infiltration. According to the Vienna classification, abnormalities are divided into the following five categories: (category 1) negative for tumor / dysplasia; (category 2) uncertain for tumor / dysplasia; (category 3) non-invasive low-grade tumor (low-grade adenoma / dysplasia); (category 4) mucosal high-grade tumor (e.g., high-grade adenoma / dysplasia), non-invasive malignancy (malignancy in situ) or suspected invasive malignancy; and (category 5) invasive tumor, intramucosal malignancy, submucosal malignancy, etc. These examples and other types of abnormalities are within the scope of the present disclosure. It should also be understood that the intelligent detector system 100 can be configured to detect other types of features of interest, including for medical and non-medical procedures.

[0083] Figure 1B An implementation for processing real-time video consistent with an embodiment of the present disclosure Figure 1A Schematic representation of an exemplary computer-implemented system of the smart detector system 100. Figure 1BAs shown, system 190 includes an imaging device 192 and an operator 191, and operator 191 operates and controls imaging device 192 via control signals sent from operator 191 to imaging device 192. By way of example, in embodiments where the video feedback includes medical video, operator 191 may be a physician or other medical professional. Imaging device 192 may include a medical imaging device, such as an endoscopic imaging device, or other medical imaging device that produces a video or one or more images of a human body / tissue / organ or a portion thereof. Imaging device 193 may be part of physician device 180 (e.g., Figure 1A 150) to generate a video stored in image source 150. Operator 191 may control image device 192 by controlling the capture or frame rate of image device 192 and / or the movement or navigation of image device 192 (e.g., through or relative to the body of a patient or individual). In some embodiments, imaging device 192 may include a swallowable capsule device or other form of capsule endoscopic device, as opposed to an endoscopic imaging device that is inserted through a body cavity.

[0084] exist Figure 1B In the example of , the image device 192 can send the captured video as a plurality of image frames directly to the computing device 193. The computing device 193 may include a memory (including one or more buffers) and one or more processors for processing the video or images, as described above and herein (e.g., see Figure 2 In some embodiments, one or more processors may be implemented not as part of computing device 193 but rather as a network (e.g., Figure 1AOne or more separate components communicating with it in a network 160 (not shown) of the computing device 193. In some embodiments, one or more processors of the computing device 193 can implement one or more networks, such as a trained neural network. Examples of neural networks include object detection networks, classification detection networks, position detection networks, size detection networks, or frame quality detection networks, as further described herein. The computing device 193 can directly receive and process multiple image frames from the imaging device 192. In some embodiments, the computing device 193 can use pre-processing and / or parallel processing and caching to process video or images in real time, and the level of such processing and caching depends on the frame rate of the received video or image and the processing speed of one or more processors or modules of the computing device 193. It should be understood that well-matched processing and caching capabilities relative to the frame rate make real-time processing and output possible. Additionally, in some embodiments, control or information signals may be exchanged between computing device 193 and operator 191 to control or direct the creation of one or more enhanced videos as output that include the original video with additional overlays (graphics, symbols, text, etc.) that provide information about the identified features of interest and other feedback generated by computing device 193 to assist the physician or operator in performing the medical procedure. With respect to the exchanged control or information signals, they may be transmitted as data to computing device 193 through imaging device 192 or directly from operator 191. Examples of control signals and information signals include signals for controlling components of computing device 193, such as the object detection network, classification detection network, position detection network, size detection network, or frame quality detection network described herein.

[0085] exist Figure 1BIn the example of FIG. 1 , the computing device 193 can use one or more modules (e.g., modules 110-140 of the smart detector system 100) to process and enhance the video received from the image device 192, and then send the enhanced video to the display device 194. The enhanced video can provide, for example, real-time feedback and reports of identified polyps and actions taken by the operator 191 during or at the end of a medical procedure (e.g., an endoscopy or colonoscopy). Video enhancement or modification may include providing one or more overlays, alphanumeric characters, text, descriptions, shapes, charts, images, animated images, and / or other suitable graphical representations in or with a video frame. Video enhancement can provide information related to features of interest, such as classification, size, actions performed, and / or location information. Additionally or alternatively, video enhancement can provide information related to one or more recommended actions identified by the computing device 193 according to the medical guideline. In order to assist the physician or operator and reduce errors, it is understood that the scope and type of information, reports, and data generated by the computing device 193 can be similar to the scope and type described above for the smart detector system 100. Therefore, reference is made to the above examples provided for the system 100.

[0086] like Figure 1B As further depicted, computing device 193 may also be configured to relay the original, unenhanced video from image device 192 directly to display device 194. For example, computing device 193 may perform direct relaying under predetermined conditions, such as when there are no overlays or other enhancements to be generated or when image device 192 is in an off state. In some embodiments, computing device 193 may perform direct relaying if operator 191 sends a command as part of a control signal to computing device 193. Commands from operator 191 may be generated by operating buttons and / or keys included on an operator device and / or input device (not shown), such as mouse clicks, cursor hovers, mouse hovers, button presses, keyboard inputs, voice commands, interactions performed in virtual or augmented reality, or any other input.

[0087] To enhance the video, computing device 193 may process the video from imaging device 192 and create a modified video stream to be sent to display device 194. The modified video may include the original image frames with the enhanced information to be displayed to the operator via display device 194. Display device 194 may include any suitable display or similar hardware for displaying video or modified video, such as an LCD display, LED display or OLED display, an augmented reality display, or a virtual reality display.

[0088] Figure 2 It shows that it can be combined with Figure 1AThe exemplary computing device 200 used in the exemplary system and other embodiments of the present disclosure. The computing device 200 can be implemented with Figure 1A The computing device 200 may be used in conjunction with one or more components of the exemplary system of the present invention, including, for example, the system 100 and devices 150, 170, and 180. In some embodiments, the computing device 200 may include multiple subsystems, such as a cloud computing system, a server, and / or any other suitable component for receiving and processing input video and images.

[0089] like Figure 2 As shown, the computing device 200 may include one or more processors 230, which may include, for example, one or more integrated circuits (ICs), including application specific integrated circuits (ASICs), microchips, microcontrollers, microprocessors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), servers, virtual servers, or other circuits suitable for executing instructions or performing logical operations as described above. In some embodiments, the one or more processors 230 may include or may be components of a larger processing unit implemented with one or more processors. The one or more processors 230 may be implemented with any combination of a general-purpose microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, a gated logic, a discrete hardware component, a dedicated hardware finite state machine, or any other suitable entity that can perform calculations or other operations on data or information.

[0090] Further Figure 2 As shown, one or more processors 230 may be communicatively connected to memory 240 via a bus or network 250. The bus or network 250 may be suitable for transmitting data and other forms of information. The memory 240 may include an internal memory portion 245, which contains instructions that perform the operations and methods described in detail herein when executed by the one or more processors 230. The memory 240 may also be used as working memory, temporary memory, and other memory or storage roles for the one or more processors 230, as appropriate. As an example, the memory 240 may be a volatile memory, such as, but not limited to, a random access memory (RAM), or a non-volatile memory (NVM), such as, but not limited to, flash memory.

[0091] The one or more processors 230 may also be communicatively connected to one or more I / O devices 210 via a bus or network 250. The I / O devices 210 may include any type of input and / or output devices or peripherals, including a keyboard, a mouse, a display device, etc. The I / O devices 210 may include one or more network interface cards, APIs, data ports, and / or other components for supporting connections to the one or more processors 230 via the network 250.

[0092] Further Figure 2 As shown, one or more processors 230 and other components (210, 240) of computing device 200 may be communicatively connected to a database or memory 220. Memory 220 may electronically store data (e.g., input video or image collections, and reports and other output data) in an organized format, structure, or file set. Memory 220 may include a database management system that facilitates data storage and retrieval. Figure 2 2 is shown as a single device, but it should be understood that the memory 220 may include multiple collocated or distributed databases or storage devices. In some embodiments, the storage 220 may be implemented in whole or in part as part of a remote network, such as a cloud storage.

[0093] One or more processors 230 and / or memory 240 may also include a machine-readable medium for storing software or instruction sets. "Software" as used herein refers broadly to any type of instruction, regardless of whether it is referred to as software, firmware, intermediate firmware, microstrip, hardware description language, or other name. Instructions may include code (e.g., in source code format, binary code format, executable code format, or any other suitable code format). When the instructions are executed by one or more processors 230, the processors may be caused to perform various operations and functions described in further detail herein.

[0094] The implementation of computing device 200 is not limited to Figure 2 The exemplary embodiment shown. The number and arrangement of components (210, 220, 230, 240) may be modified and rearranged. In addition, although Figure 2 200 may be in electronic communication with other networks, including the Internet, local area networks, wide area networks, metropolitan area networks, and other networks capable of communicating between elements of the computing architecture. In addition, the computing device 200 may retrieve data or other information described herein from any source, including the memory 220, and from a network or other database. In addition, the computing device 200 may include one or more machine learning models for implementing the neural networks and other modules described herein, and may retrieve or receive weights or parameters of the machine learning models, training information or training feedback, and / or any other data and information described herein.

[0095] Figure 3A is a block diagram of an exemplary local spatiotemporal processing module consistent with embodiments of the present disclosure. Figure 3A The embodiments can be used to implement Figure 1A The local spatiotemporal processing module 110 of the exemplary smart detector system 100 or other computer-implemented system, such as Figure 1B The computing device 193 in FIG. Figure 3A As shown, the local spatiotemporal processing module 110 includes multiple components, including a sampler 311, an encoder 312, a recursive neural network (RNN) 313, a temporal convolutional network (TCN) 314, a quality network 315, and a segmentation network 316. These components can be implemented in any appropriate combination of hardware, software, and firmware, and are used to select and process image subsets. The local spatiotemporal processing module 110 can iteratively apply various networks to the entire input video or image set (with T image frames) using an input batch of N image frames (where N≥1). Such image frames can be continuous or sampled at a fixed rate from the input video or image set.

[0096] The sampler 311 may select image frames for processing by other components of the local spatiotemporal processing module 110. In some embodiments, the sampler 311 may cache the input video or image set at a set period and extract the image frames in the cache as an image subset for processing by the module 110. In some embodiments, the sampler 311 may allow the configuration of the number of frames or the size of the image subset for selection for processing by the local spatiotemporal processing module 110. For example, the sampler 311 may be configured to receive user input for setting or tuning the number of frames or the size of the image subset for processing. Additionally or alternatively, the sampler 311 may be configured to automatically select the number of image frames or the size of the image subset based on other factors, such as the requested feature of interest for processing and / or characteristics related to the feature of interest. In some embodiments, the number or size of sampled images may be based on the frame rate of the video (24, 30, 60 and 120 frames / second). For example, the sampler 311 may periodically cache the real-time video stream received by the smart detector system 100 at a set period to extract images from the cached video. As another example, a stream of images from a pill box or other imaging device may be buffered, and images for processing by the system 100 may be extracted from the buffer. In some embodiments, the sampler 311 may selectively sample images based on other factors such as video quality, length of the video, characteristics related to the requested feature of interest, and / or features of interest. In some embodiments, the sampler 311 may sample image frames based on components of the local spatiotemporal processing module 110 involved in performing a task requested by a user of the smart detector system 100. For example, an encoder 312 using a 3D encoder network may require multiple images to create a three-dimensional structure of the content to be encoded.

[0097] The encoder 312 can determine the presence of characteristics associated with each feature of interest as part of a task requested by a user of the smart detector system 100. For image analysis, the encoder 312 can be implemented using a trained convolutional neural network. The smart detector system 100 can include a two-dimensional encoder and a three-dimensional encoder that contains a non-local layer as the encoder 312. The encoder 312 can be composed of multiple convolution residuals and fully connected layers. Depending on the features and characteristics of interest to be detected, the encoder 312 can select a two-dimensional or three-dimensional convolutional encoder. The encoder 312 can be trained to detect characteristics in the image that are required to detect the requested features of interest in the image frame. As disclosed herein, the smart detector system 100 can use the encoder 312 to process the image and detect the desired characteristics associated with the features of interest. The smart detector system 100 can determine the desired characteristics based on the trained network of the encoder 312 and the past determination of the features of interest.

[0098] like Figure 3AAs shown, the local spatiotemporal processing module 110 may also be equipped with a recurrent neural network (RNN) 313. The RNN 313 may cooperate with the encoder 312 to determine the presence of a desired characteristic associated with the feature of interest. In some embodiments, a temporal convolutional network (TCN) 314 may also be used to assist in detecting such characteristics in each image frame of the input video. A TCN is a specialized convolutional neural network that can process a sequence of images, such as from a source (e.g., Figure 1A Image source 150 or Figure 1B The TCN may work on a subset of images of all image frames of the input video that are cached for processing by the local processing module 110, or on a subset of all images of the input video that are processed by the global processing module 120. The TCN may use causal convolutions in a fully connected convolutional network to process sequential data.

[0099] RNN 313 is an artificial neural network with internal feedback connections and internal memory states for determining spatiotemporal information in image frames. In some embodiments, RNN 313 may include a local layer to improve its ability to separate spatiotemporal information in space and / or time in the cached image frames selected by sampler 311. RNN 313 can be configured to associate a score (e.g., between 0-1) for each desired characteristic associated with the requested feature of interest. The score indicates the likelihood of the desired characteristic being present in the image, with 0 being the least likely and 1 being the most likely.

[0100] like Figure 3A As shown, the local spatiotemporal processing module 110 may include additional components, such as a quality network 315 and a segmentation network 316, to further help identify the features required to detect features of interest in the processed image. For example, the quality network 315 can be implemented as a neural network consisting of several convolutional layers and several fully connected layers that improve the final score assigned to the image frame. For example, the quality network 315 can filter out image frames with low feature scores. The quality network 315 can also generate a feature vector based on the determined features for each image frame. Each feature vector can provide a quality score, which is expressed as an ordinal number [0, R] indicating the quality of the frame image, where 0 is the minimum quality and R is the maximum quality. The quality network 315 can automatically set a quality score of 0 for those features that are not detected in the image.

[0101] The segmentation network 316 can process images to calculate a segmentation mask for each input image to extract segments of the image, which include features related to the feature of interest. The segmentation mask can be a pixel-scale binary mask having the same or less resolution than the input image. The segmentation network 316 can be implemented as a deep convolutional neural network including multiple convolutional residual layers and multiple skip connections. The number of layers and the type of layers included in the segmentation network can be based on the feature or requested feature of interest. In some embodiments, a single model with multiple layers can handle the tasks of the encoder 312 and the segmentation network 316. For example, a single model can be a U-Net model with a ResNet encoder.

[0102] By way of example, the segmentation network 316 may take as input an image having dimensions W×H and return a segmentation mask represented by a matrix having dimensions W'×H', where W' is less than or equal to W and H' is less than or equal to H. Each value in the output matrix represents the probability that a particular image frame coordinate contains a feature associated with the requested feature of interest. In some embodiments, the smart detector system 100 may generate multiple output matrices for multiple features of interest. The multiple matrices may vary in size.

[0103] Figure 3B is a block diagram of an exemplary global spatiotemporal processing module consistent with embodiments of the present disclosure. Figure 3B The embodiments can be used to implement Figure 1A The global spatiotemporal processing module 120 of the exemplary smart detector system 100 or other computer-implemented system, such as Figure 1B The computing device 193 in the network 160. The global spatiotemporal processing module 120 can modify or refine the output obtained from the local spatiotemporal processing module 110. Therefore, the global spatiotemporal processing module 120 can be configured to process the complete set of images together after the local spatiotemporal processing module 110 processes all image subsets of the input video obtained from, for example, the image source 150 through the network 160. By way of example, the global spatiotemporal processing module 120 processes the output of the complete set of images processed by the local spatiotemporal processing module 110 to modify its output by refining and filtering out images with determined characteristics or features of interest. In some embodiments, the global spatiotemporal processing module 120 can refine the quality scores and segmentation masks generated by the quality network 315 and the segmentation network 316.

[0104] The global spatiotemporal processing module 120 may refine the scores of the determined features using one or more non-causal temporal convolutional networks (TCNs) 321. The global spatiotemporal processing module 120 may process the outputs of all images processed by the local spatiotemporal processing module 110 using a dilated convolutional network including a TCN 321. Such a dilated convolutional network helps increase the receptive field without increasing the network depth (number of layers) or kernel size, and can be used for multiple images together.

[0105] As further disclosed herein, TCN 321 may examine an entire time series of K×T′ features extracted using local spatiotemporal processing module 110. TCN 321 may take an input matrix of features of K×T′ dimensions and return one or more multiple time series of scalar values ​​of length T″.

[0106] The global spatiotemporal processing module 120 may use one or more signal processing techniques such as low-pass filters and Gaussian smoothing to refine the quality scores generated by the quality network 315. In some embodiments, the global spatiotemporal processing module 120 may use a cascade of morphological operations to refine the segmentation mask generated by the segmentation network 316. The global spatiotemporal processing module 120 may refine the binary mask used for segmentation by combining the RNN 313 and the causal TCN 314 using prior information about the shape and distribution of the determined features between the input images identified by the encoder 312.

[0107] The TCN 321 may work on a complete video or set of input images and therefore needs to wait on the local spatiotemporal processing module 110 to finish processing a single image frame. To accommodate this requirement, the TCN 321 may be trained after the network training in the local spatiotemporal processing module 110. The number of layers and architecture of the TCN 321 depends on the task requested by the user of the smart detector system 100 to detect certain features of interest. The TCN 321 may be trained based on the requested features of interest to be assigned to the system. The training algorithm for the TCN 321 may tune the parameters of the TCN 321 for each such task or feature of interest.

[0108] By way of example, the smart detector system 100 may first calculate the K-dimensional time series of each video in the training set 415 using the local spatiotemporal processing module 110, and apply gradient descent-based optimization to estimate the TCN 321 parameters to minimize the loss function L(s, s') to train the TCN 321, where s is the output time series of the estimated scores and s' is the true time series. The smart detector system 100 may calculate the distance between s and s' using, for example, mean squared error (MSE), cross entropy, and / or Huber loss.

[0109] Similar to the training process for other neural networks, data augmentation, learning rate tuning, label smoothing, and / or batch training may be used when training TCN 321 to improve its capabilities and results.

[0110] The smart detector system 100 can be adapted to specific requirements by adjusting the hyperparameters of the components 110-130. In some embodiments, the smart detector system 100 can modify the standard architecture of the circuit to process the input video or image set by adding or removing components 110-130 or certain parts of the components 110-130. For example, if the smart detector system 100 needs to process a very local task, it can discard the use of TCN 321 in the global spatiotemporal processing module 120 to avoid any global spatiotemporal processing of the output generated by the local spatiotemporal processing module 110. As another example, if the user of the smart detector system 100 requests a diffusion task, the RNN 313 and / or TCN 314 can be removed from the local spatiotemporal processing module 110. The smart detector system 100 can remove the global spatiotemporal processing module 120 or some RNNs 313 and TCN 314 of the local spatiotemporal processing module 110 by deactivating the relevant network from the circuit used to process the input video or image to detect the requested pathology.

[0111] Other arrangements or implementations of the system are also possible. For example, when the requested task for detecting features of interest does not process a focal object in an image frame of the image, the segmentation network 316 may be unnecessary and discarded from the pipeline. As another example, the quality network 315 may be unnecessary and deactivated when all image frames are considered useful. For example, when the frame rate of the input video is low or has too many image frames with errors, the smart detector system 100 can avoid further filtering the image frames by using the quality network 315. As will be understood from the present disclosure, the smart detector system 100 can pre-process and / or sample the input video or image to determine the components that need to be activated and trained as part of the local spatiotemporal processing module 110 and the global spatiotemporal processing module 120.

[0112] Figure 3C is a block diagram of an exemplary time series analysis module consistent with embodiments of the present disclosure. Figure 3C The embodiments can be used to implement Figure 1A The time series analysis module 130 of the exemplary smart detector system 100 or other computer-implemented system, such as Figure 1BThe time series analysis module 130 may use the scores, quality values, and segmentation maps of each image frame processed by the local spatiotemporal processing module 110 as input to generate a final output score for each image. In some embodiments, after the medical procedure is completed, the time series analysis module 130 may use the scores, quality values, and / or segmentation maps of the images generated by the local spatiotemporal processing modules to generate a panel or other display with a summary of the quality scores of all images. The components of the time series analysis module 130 may be used to generate different output scores and values ​​that are presented as an aggregate summary of the images processed by the smart detector system 100. Figure 3C As shown, components of the time series analysis module 130 may include an event detector 331, a frame selector 332, an object descriptor 333, and a temporal segmenter 334 to help generate a final output score for an image of an input video.

[0113] The event detector 331 can determine the start and stop times in the input video of the event associated with the requested feature of interest. In some embodiments, the event detector 331 determines the start image frame and the end image frame in the input video of the event associated with the requested feature of interest. In some embodiments, the start and stop times or image frames of the event can be superimposed.

[0114] The start and stop times of an event may be the start and end of a portion of the input video where some characteristic associated with the feature of interest is detected. The start and stop times of the video may be estimates due to missing image frames analyzed from the local spatiotemporal processing module 110. The event detector 331 may output a list of pairs (t, d) where t is a time instance and d is a description of the event detected at that time. Various events may be identified based on different features of interest processed by the smart detector system 100.

[0115] The portions of the input video identified from the events may include portions of an organ scanned by a healthcare professional or other operator to generate the input video as part of a medical procedure. For example, a medical procedure (e.g., a colonoscopy) may include events configured for various portions of the colon (e.g., the ascending colon, the transverse colon, or the descending colon).

[0116] The time series analysis module 130 can provide a summary report of events for different portions of the video representing different portions of the medical procedure. The summary report can include, for example, the length of the video or time taken to complete a scan of a portion of an organ associated with the event, which can be listed as the exit time. The event detector 331 can help generate a summary report of different portions of the medical procedure including events associated with the feature of interest.

[0117] The time series analysis module 130 can present summary reports of different portions of a medical procedure video (e.g., a colonoscopy video) on a panel or other display that displays, for example, a pie chart, where the amount of video requires different actions to be taken on portions of the video or portions of the organs represented by the portions of the video, such as careful secondary inspection, performing a biopsy, or removing a lesion. In some embodiments, the panel can include quality summary details of the events identified by the event detector 331 in a color-coded manner. For example, the panel can include red, orange, and green buttons or other icons to identify the video quality of the portion of the medical procedure representing the event. The panel can also include summary details of the overall video representing the entire medical procedure, with the same level of information as provided for the individual portions of the medical procedure.

[0118] In some embodiments, the summary report generated by the time series analysis module 130 may identify one or more frames to more closely examine the portion and / or resolve other issues. The summary report may also indicate a percentage of the video for additional action (e.g., secondary inspection). The time series analysis module 130 may use the frame selector 332 to identify specific frames of the video or percentage of the video for additional action.

[0119] The frame selector 332 may retrieve image frames in the input video based on the characteristics and scores generated by the local spatiotemporal processing module 110. In some embodiments, the frame selector 332 may also select image frames using a quality value provided by a user. The frame selector 332 may select image frames based on their relevance to features and / or characteristics of interest requested by a user of the smart detector system 100.

[0120] In some embodiments, the summary report generated by the time series analysis module 130 may include one or more image frames identified by the frame selector 332. The image frames presented in the report may be enhanced to display markers applied to one or more portions of the frames. In some embodiments, the markers may identify features of interest, such as lesions or polyps in the image frames. For example, a colored bounding box may be used as a marker around the feature of interest (e.g., see Fig.10 and a green bounding box applied to the image frame shown in the figure). In some embodiments, different markers (including different combinations of shapes and / or colors) may be used to indicate different features of interest. For example, an image frame may be enhanced to include one or more markers in the form of bounding boxes of different colors representing different features of interest identified by the smart detector system 100.

[0121] The object descriptor 333 may merge image frames of the input video that include matching features from the requested feature of interest. The object descriptor merges the image frames based on the temporal and spatial coherence information provided by the local spatiotemporal processing module 110. The object descriptor 333 output may include a set of objects described using a set of attributes. The set of attributes may include a timestamp of the image frame relative to other image frames of the input video. In some embodiments, the set of attributes may include statistics about the estimated score and location of the detected feature or requested feature of interest in the image frame.

[0122] The time segmenter 334 segments the input video into time intervals. The time segmenter 334 can be segmented based on the relevance of the task to determine the requested feature of interest. The time segmenter 334 can output a label for each image frame of the input video in the form of {L_i}. The output label can indicate the presence and probability of the requested feature of interest in the image frame and the position within the image frame. In some embodiments, the time segmenter 334 can output a separate label for each feature of interest in each image frame.

[0123] In some embodiments, the time series analysis module 130 can generate a panel or other display including a quality score of a medical procedure performed by a physician, a medical professional, or other operator. In order to provide a quality score, the time analysis module 130 can include a machine learning model trained based on videos of medical procedures performed by other physicians and operators with different inspection performance behaviors. In addition, the machine learning model can be trained to identify video clips where the inspection behavior of medical professionals indicates that additional inspections are required. For example, an endoscopist carefully explores the surface of the colon / small intestine instead of spending time cleaning the colon or performing operations such as surgery or navigation, which will indicate the need for additional inspection of the surface of the small intestine. The machine learning model used by the time series analysis module 130 can learn about specific activities of medical professionals, such as careful exploration, based on the amount of time spent on a specific part of a medical procedure representing a specific part of an organ, the number of pictures taken, and / or the number of repeated scans. In some embodiments, the machine learning model can learn about medical professional behavior based on the amount of annotations or markers in the form of certain areas of an image frame added to a video or video.

[0124] In some embodiments, the time series analysis module 130 can use information about the time spent performing specific actions (e.g., careful exploration, navigation, cleaning, etc.) to generate a summary report of the quality score of the medical professional's behavior. In some embodiments, the percentage of the total time of the medical procedure for a specific action can be used to calculate the quality score of the medical procedure or a portion of the medical procedure. The time series analysis module 130 can be configured to generate a quality summary report of the medical professional's behavior based on the configuration of the smart detector system 100 to include the actions performed by the medical professional as features of interest.

[0125] To generate a panel with the summary scores described above, the time series module can utilize in combination an event detector 331, a frame selector 332, an object descriptor 333, and a time segmenter 334. The panel can include one or more frames from a medical procedure selected by the frame selector 332, as well as information about the total time spent on the medical procedure and the time spent examining portions having pathology or other features of interest. A statistical quality score summary describing the health care professional's behavior can be calculated for the entire medical procedure (e.g., an entire colon scan) and / or for one or more portions of an identified anatomical region (e.g., colon segments such as the ascending colon, transverse colon, and descending colon).

[0126] The time series analysis module 130 may use the event detector 331, the frame selector 332, the object descriptor 333, and the time segmenter 334 to generate aggregate information about different features of interest, such as different regions of an organ captured during a medical procedure, the presence of each pathology, and / or the actions of a healthcare professional performing the medical procedure. For example, aggregate information may be generated based on a list of various pathologies in different regions using the object descriptor 333, frames showing the pathology selected by the frame selector 332, and the identified amount of time spent in the region of each pathology and the actions of the healthcare professional determined by the event detector 331.

[0127] In some embodiments, the time series analysis module 130 can generate a summary of the input video processed by the local spatiotemporal processing module 110 and the global spatiotemporal processing module 120. The summary of the input video may include segments of the input video extracted and combined into a summary of the input video with interesting features. In some embodiments, the user can choose whether to watch only the summary video or expand each interval of the video discarded by the module. The time segmentation 334 of the time series analysis module 130 can help extract the part of the input video with interesting features. In some embodiments, the time series analysis module 130 can generate a video summary by selecting relevant frames to generate a variable frame rate video output. The frame selector 332 of the time series analysis module 130 can help select and discard frames in the output video summary. In some embodiments, the time series analysis module 130 can provide additional metadata to the input video or video summary. For example, the time series analysis module 130 can color-code the timeline of the input video, where interesting features are present in the input video. The time series analysis module 130 can use different colors to highlight the timeline with different interesting features. In some embodiments, the part of the output video summary with interesting features can include text and graphics superimposed on the output video summary.

[0128] In some embodiments, to maximize performance, modules 110 - 130 may be trained to select optimal parameter values ​​for the neural network in each of modules 110 - 130 .

[0129] Figure 3A The components of the local spatiotemporal processing module 110 shown may include a neural network that is trained before being used to process images and detect characteristics associated with features of interest. The neural network in the local spatiotemporal processing module 110 may be trained based on a video data set that includes the following three subsets: a training set 415, a validation set 416, and a test set 417 (see Figure 4A ). A training subset for the smart detector system 100 may require a labeled video set. For example, a labeled video set set may include a target score assigned to the video processing by a component of the smart detector system 100. The labeled video set may also include the location of detectable features and the value of each feature in each image frame of the video set. Both the labels and the input video set may be used for training purposes. In some embodiments, the labels of a subset of the video set may be used to determine the labels of other subsets used to train the neural network in the components 110-130 of the smart detector system 100.

[0130] During the training process, the smart detector system 100 can sample from a training data set image or a buffer of images processed by the neural network in the components 110-130 of the smart detector system 100, and update their parameters through error back propagation. The smart detector system 110 can use the validation set 416 of the video set to control the convergence of the true value y' of the desired feature and the encoder 312 output value y. The smart detector system 100 can use the test set 417 of the video set to evaluate the performance of the encoder 312 to determine the feature values ​​in the image frames of the training subset of the video set. The smart detector system 100 can continue training until the true value y' converges with the output value y. The smart detector system 100 can complete the training process and remove the temporary fully connected network when convergence is reached. The smart detector system 100 determines the encoder 312 for the latest value of the parameter.

[0131] Figure 4A It is consistent with the embodiment of the present disclosure. Figure 3A Flowchart of an exemplary training of an encoder component of a local spatiotemporal processing module of . The encoder (e.g. Figure 3A The encoder 312) component takes a single image frame or a small buffer of N image frames as input and produces an M-dimensional feature vector as output. Figure 4A As shown, the smart detector system 100 can train the encoder 312 by adding a temporary network. The temporary network can be a fully connected network (FCN) 411, which is added as an additional layer at the end of the encoder 312 to train the encoder 312. The FCN 411 can take an input feature vector for each image frame of the input generated by the encoder 312 and return a single floating value or a one-hot vector y. The smart detector system 100 can use a loss function 413 to evaluate the convergence of the true value y' and the output y of the encoder 312. The loss function can be an additional layer added as the last layer of the encoder 312. The loss function 413 can be expressed as L(y, y'), which indicates the distance between the true value of the feature y' and the output y generated by the encoder 312 for the image frame of the input video. The smart detector system 100 can use mean squared error (MSE), cross entropy, or Huber loss as the loss function 413 to train the encoder 312 using the FCN 411.

[0132] In some embodiments, the temporary network can be a decoder network 412 used by the smart detector system 100 to train the encoder 312. The decoder network 412 can be a convolutional neural network that maps each feature vector estimated by the encoder 312 to a large matrix (I_out) of the same dimension as the image frame (I_in) of the input video. The decoder network 412 can use L (I_out, I_in) as a loss function 413 to calculate the distance between two images (or a buffer of N images). The loss function 413 used with the decoder network 412 can include mean square error (MSE), structural similarity (SSIM), or L1 norm. The decoder network 412 used as a temporary network for training the encoder 312 does not need to determine the true value for the training / validation / test subsets 415-417 of the video set. The smart detector system 100 training the encoder 312 using the decoder network 412 as a temporary network can control the convergence with the validation set 416, and use the test set 417 to evaluate the expected performance of the encoder 312. The smart detector system 110 may discard or deactivate the decoder network 412 after completing the encoder 312 training.

[0133] In the training method using the fully connected network 411 and the decoder network 412, the encoder 312 and other components of the smart detector system 100 can use techniques such as data augmentation, learning rate tuning, label smoothing, mosaic, MixUp and CutMix data augmentation and / or batch training to improve the training process of the encoder 312. In some embodiments, the neural network in the smart detector system 100 may suffer from class imbalance, and a self-organizing weighted loss function and importance sampling can be used to avoid prediction bias of the majority class.

[0134] Figure 4B It shows Figure 3A A flowchart of an exemplary training of one or more neural network components of an exemplary local spatiotemporal processing module. Figure 4B The exemplary training of can be used to train, for example, the recurrent neural network (RNN) 313 and the temporal convolutional network (TCN) 314 of the local spatiotemporal processing module 110 consistent with embodiments of the present disclosure. Training deep neural networks (e.g., RNN 313 and TCN 314) in the smart detector system 100 may require preparing a set of annotated videos or images and loss functions during the training process. During the training process, the smart detector system 100 may use a gradient descent-based optimization algorithm to adjust network parameters.

[0135] The smart detector system 100 can train the RNN 313 and TCN 314 using the output of the previously trained encoder 312. The input to the RNN 313 and TCN 314 can be an M-dimensional feature vector at each moment output by the encoder 312. The RNN 313 and TCN 314 aggregate multiple feature vectors generated by the encoder 312 by caching the feature vectors generated by the encoder 312. The smart detector system 100 can train the RNN 313 and TCN 314 by feeding a sequence of continuous image frame encoders 312 and passing the generated feature vectors to the RNN 313 and TCN 314. For B image sequences (or cached image sets), the encoder 312 generates a B vector of M encoded features and sends it to the RNN 313 or TCN 314 to generate a B vector of K features.

[0136] The smart detector system 100 can train the RNN 313 and TCN 314 by including a temporary fully connected network (FCN) 411 at the end of the RNN 313 and TCN 314. The FCN 411 converts the K-dimensional feature vector generated by the RNN 313 or TCN 314 into a one-dimensional score and compares it with the true value in the loss function to modify the parameters until there is convergence between the output vector and the true value vector. In some embodiments, the smart detector system 100 improves the RNN 313 and TCN 314 by using data enhancement, learning rate tuning, label smoothing, batch training, weighted sampling and / or importance sampling as part of training the RNN 313 and TCN 314.

[0137] Figure 4C It shows Figure 3C Flowchart of exemplary training of the mass network and segmentation network components of the local spatiotemporal processing module. Figure 4C The exemplary training of can be used to train the quality network 315 and segmentation network 316 of the exemplary local spatiotemporal processing module 110 consistent with embodiments of the present disclosure. The smart detector system 100 can train the quality network 315 similar to the training of the encoder 312, but does not require the temporary network ( Figure 4AThe quality network 315 outputs a scalar value q representing the quality of the image. The smart detector system 100 can train the quality network 315 by comparing its output quality score q with the true quality score q' (which is associated with each image frame of the training set 415 of the video set) until the difference between the values ​​is minimized. The smart detector system 100 can use a loss function 413 denoted as L(q, q') to minimize the difference between the true value q' and the output quality score q and adjust the parameters of the quality network 315. The smart detector system 100 can use MSE, L1 norm as the loss function 413 to train the quality network 315. The smart detector system 100 can use data augmentation, learning rate tuning, label smoothing and / or batch training techniques to improve the training results of the quality network 315.

[0138] The smart detector system 100 can use one or more individual images or a small buffer of size N to train the segmentation network 316. The buffer of size N can be based on Figure 4A The number of images considered by the encoder 312 trained in the training. The smart detector system 100 requires annotated ground truth maps as part of the training set 415 to train the segmentation network 316. The smart detector system 100 can use a loss function 413, which is denoted as loss L(m, m'), which defines the distance between the map m estimated by the segmentation network 316 and the true map m'. The smart detector system 100 can compare the predicted map m and the true map m' by using, for example, pixel-wise MSE and L1 loss functions as well as dice scores and smoothed dice scores.

[0139] In some embodiments, the smart detector system 100 may use data augmentation, such as self-organizing morphological operations and affine transformations with each image frame in the input video and a mask generated for each image frame, learning rate tuning, label smoothing, and / or batch training to improve the results of the segmentation network 316.

[0140] Figure 4D It shows Figure 3B Flowchart of an exemplary training of a global spatiotemporal processing module. Figure 4D The exemplary training of can be used to train the global spatiotemporal processing module 120 consistent with the embodiments of the present disclosure. Figure 4D As shown, the smart detector system 100 can train the global spatiotemporal processing module 110 by using the output of the local spatiotemporal processing module 110. As will be understood from this disclosure, the local spatiotemporal processing module 110 needs to be trained before it is used to train the global spatiotemporal processing module 120. The local spatiotemporal processing module 110 is trained by training each of its components separately, as described above. FIG. 4A to FIG. 4C Described.

[0141] The temporal convolutional network (TCN) 321 of the global spatiotemporal processing module 120 can access the entire time sequence of features T'×K extracted by the local spatiotemporal processing module 110 operating on T' image frames to generate a feature vector of 1×K dimensions. The global spatiotemporal processing module 120 takes the entire matrix T'×K of features as input and returns a time sequence of scalar values ​​of length T". The smart detector system 100 can train the global spatiotemporal processing module 120 by training the TCN 321.

[0142] The smart detector system 100 trains the TCN 321, and the global spatiotemporal processing module 120 may take into account the number of processing layers and its architectural structure of the TCN 321. The number of layers and connections varies depending on the task used to determine features of interest and needs to be adjusted for each task.

[0143] The smart detector system 100 trains the global spatiotemporal processing module 120 by computing a K-dimensional time series of scores for the image frames of each video in the training set 415. The smart detector system 100 computes the time series scores by providing the training set 415 videos as input to the previously trained local spatiotemporal processing module 110 and providing its output to the global spatiotemporal processing module 120. The smart detector system 100 can estimate the network parameters of the TCN 321 neural network using gradient descent-based optimization. The gradient descent-based optimization can minimize the distance between the time series scores output by the global spatiotemporal processing module 120 and the true time series scores '. The loss function 413 used to train the global spatiotemporal processing module 120 can be a mean squared error, a cross entropy, or a Huber loss.

[0144] In some embodiments, the smart detector system 100 may use data augmentation, learning rate tuning, label smoothing, and / or batch training techniques to improve the results of the trained global spatiotemporal processing module 120 .

[0145] Figure 5A and Figure 5B is a schematic representation of a circuit constructed with components of an exemplary intelligent detector system for processing an input video or image set. By way of example, Figure 5A and Figure 5B The lines in can be Figure 1A The components of the smart detector system 100 are used to process input video. The circuitry for processing input video using the modules of the smart detector system 100 may vary in structure based on the type of input video to be processed and the features of interest requested. The circuitry may include all or some components of each module (e.g., Figure 3A encoder 312).

[0146] like Figure 5A As shown, circuit 500 includes components of smart detector system 100 to process input video 501 and determine whether a user or physician (e.g., by Figure 1A The circuit 500 includes components of the local spatiotemporal processing module 110 and the global spatiotemporal processing module 120 to generate matrices 531 and 541 of scores of the determined features and the requested features of interest in each image frame. The circuit 500 also includes a time series analysis module 130 for determining the features of interest using the spatiotemporal information of the features present in the matrices 531 and 541.

[0147] The local spatiotemporal processing module 110 may output a K×T′ matrix 531 of feature scores. T′ is the number of frames of the input video 501 that are iteratively analyzed by the local spatiotemporal processing module 110. The local spatiotemporal module 110 generates a vector of size K of feature scores for each analyzed frame of the T′ frames. The size K may match the number of features of interest requested by a user of the smart detector system 100. The local spatiotemporal processing module 110 may process the input video 501 using a sampler 311 to retrieve some or all of the T image frames. The T′ frames analyzed by the components of the local spatiotemporal processing module to generate feature scores may be less than or equal to the total T image frames of the input video 501. The sampler 311 may select T′ frames to be analyzed by other components 312 and 315-317. In some embodiments, the RNN 313 and TCN 314 may generate scores for only T′ image frames of the sampled frames. The networks 313-314 may include T′ image frames based on the presence of at least one feature of the requested feature of interest. The local spatiotemporal processing module uses only one set of networks 313 or 314 to process the image frame and generate a matrix 531 of feature scores.

[0148] The local spatiotemporal processing module generates a matrix 531 of characteristic scores for T′ image frames by examining each image frame individually or in combination with a subset of image frames of the input video 501 buffered and provided by the sampler 311 .

[0149] The local spatiotemporal processing module 110 may generate additional matrices 532-534 of scores using networks 315-317. The quality network 315 may generate a quality score for each image frame considered by the sampler 311 for determining characteristics associated with features of interest in each image frame. Figure 5AAs shown, the quality network 315 can generate a matrix 532 of quality scores. The matrix 532 can be a 1×T” vector of quality scores of T” frames analyzed by the quality network 315. The quality network 315 can analyze the image frames extracted by the sampler 311 to generate quality scores of T” image frames. In some embodiments, T” can be less than the total number of T frames of the input video 501. The quality network 315 can only process T” frames whose quality scores are above a threshold.

[0150] The segmentation network 316 can generate a matrix 533 of segmentation masks by processing T"' image frames of the input video 501. The dimension of the matrix 533 is W'×H'×T"', including T"' masks of height H' and width H'. In some embodiments, the width W' and height H' of the segmentation mask can be smaller than the size of the processed image frame. The segmentation network 316 can analyze the image frames extracted by the sampler 311 to generate segmentation masks for T"' image frames. In some embodiments, T"' can be less than the total number of T frames of the input video 501. If the T"' frames include at least some features of interest or requested characteristics, the segmentation network 316 may only process T"' frames with segmentation masks.

[0151] like Figure 5AAs shown, the global spatiotemporal processing module 120 processes the matrix 531 of K×T' scores of the features output by the local spatiotemporal processing module 110 to generate a modified matrix 541 of feature scores. The global spatiotemporal processing module 120 checks the feature scores of all analyzed T' image frames by processing the matrix 531 of features together. The TCN 321 in the global spatiotemporal processing module 120 can process the matrix 531 of feature scores to generate a matrix 526 of scores of dimension 1×T'. The TCN 321 generates the matrix 526 of scores by combining the scores of T' image frames represented by vectors of size K. The global spatiotemporal processing module 120 can use a post-processor 522 to remove any outliers in the matrix 526. The post-processor 522 can use standard signal processing techniques such as low-pass filters and Gaussian smoothing to remove outliers. The global spatiotemporal processing module 120 outputs a matrix 541 of floating scores of dimension 1×U'. The dimension of U' can be less than or equal to T'. The post-processor 522 may filter some of the T' image frame scores to generate a refined matrix 541 of scores. In some embodiments, the U' dimension may be greater than T' obtained by upsampling the input video to increase the number of image frames of the input video (e.g., video 501). The global spatiotemporal processing module 120 may include an upsampling module to increase the number of frames. If the number of image frames with quality scores is below a threshold, the global spatiotemporal processing module 120 may upsample the video 501. The global spatiotemporal processing module 120 may upsample image frames with high quality scores based on the quality network 315 determined by the local spatiotemporal processing module 110. In some embodiments, the video 501 may be upsampled before being processed by the global spatiotemporal processing module 120. For example, the sampler 311 may upsample the input video 501 to create additional image frames.

[0152] The global spatiotemporal processing module 120 may use post-processors 523 - 525 to refine the matrices 532 - 534 of additional scores and details used in determining the requested features of interest to generate matrices 542 - 544 .

[0153] By way of example, the post-processor 523 refines the quality score matrix 532 using one or more standard signal processing techniques such as a low-pass filter and Gaussian smoothing. The post-processor 523 outputs a matrix 542 of refined scores of dimension 1×U″. In some embodiments, the value U″ may be different from the value T″. For example, if certain image frames with low quality scores are ignored by the post-processor 523, U″ may be less than T. Alternatively, when the video 501 is upsampled to produce more image frames and image frames with higher resolution, U″ may be greater than T″.

[0154] The post-processor 524 may refine the segmentation mask matrix 533 using a cascade of morphological operations that exploit prior information about the shape and distribution of each feature of interest. The post-processor 524 may output a matrix 543 of dimensions W'×H'×U"'. In some embodiments, the dimensions of U"' may be different from T"'. For example, if the post-processor 524 ignores certain image frames with low quality scores, U"' may be less than T"'. Alternatively, when the video 501 is upsampled to produce more image frames and image frames with higher resolution, U"' may be greater than T"'.

[0155] like Figure 5A As shown, the time series module 130 of circuit 500 can obtain output matrices 541-544 from the global spatiotemporal processing module 120 to generate numerical values ​​indicating the location in the input video and the location in each image frame of the input video of the requested feature of interest. The time series module 130 can use the feature scores in matrix 541 and the quality score matrix 542 to select the image frame that best represents the presence of each feature of interest. In some embodiments, the time series module 130 can use the spatiotemporal information of the features in the image frames in matrix 541 to determine the intervals in the input video 501 that include the features of interest.

[0156] like Figure 5B As shown, line 502 illustrates an alternative architecture that does not include additional components such as networks 315-317 (eg, Figure 5A ) and post-processors 523-525 (also as Figure 5A ). Line 502 can still produce the same feature score matrices 531 and 541 as outputs of the local spatiotemporal processing module 110 and the global spatiotemporal processing module 120. The time series module 130 takes the matrix 541 as input to produce values ​​identifying features of interest in the input video 501.

[0157] Fig. 6A and Figure 6B An example of a method for using a smart detector system (e.g., Figure 1AThe exemplary system 100 of the smart detector system 100 can be configured and managed as lines to process image data for different tasks. The task manager 140 can maintain different line architectures and manage the data flow across different modules in the lines. In some embodiments, the smart detector system 100 can be used to determine various features of interest requested by different users from the same input video as different tasks. In this case, the neural network in the smart detector system 100 can be trained for different tasks to determine the features of interest associated with each task. The smart detector system 100 can also be trained for different types of input videos generated by different medical devices and / or other imaging devices to determine different features of interest.

[0158] The task manager 140 can maintain separate circuits for each task and train them independently. Fig. 6A As shown, the smart detector system 100 can use modules to generate two separate circuits 610 and 620, and train them to work on separate tasks 602 and 603 to process the input video 601 to detect different features of interest. In some embodiments, the circuits 610 and 630 are pre-trained to process different tasks. In addition, the smart detector system 100 can instantiate circuits by retrieving relevant pre-trained modules for processing the input video 601. For example, the task manager 140 can include a local spatiotemporal module 611, a global spatiotemporal module 612, and a time series module 613 in the circuit 610 to process the video 601 to determine the features of interest requested as part of the task 602. Similarly, the circuit 620 can be constructed using a local spatiotemporal processing module 621, a global spatiotemporal processing module 622, and a time series module 623 to process the video 601 to determine the features of interest as part of the task 603. Maintenance of multiple circuits facilitates easy expansion to multiple tasks, but may result in redundant processing of image data by certain components. In the following embodiments, in Figure 6B An efficient alternative to a hybrid line architecture with a partially shared set of components is described in the example of .

[0159] Figure 6B An alternative circuit 650 is shown with shared modules of the smart detector system 100 between tasks. The smart detector system 100 shares modules between different tasks by sharing some or all components of each module. The smart detector system 100 may share those components of the circuit that are less dependent on the requested task to process the image data.

[0160] The samplers 311 and the quality network 315 can be dependent on the input image data and work on the image data in the same manner regardless of the requested features of interest. Thus, in line 650, the component samplers 631 and the quality network 635 of the local spatiotemporal processing module 630 that are dependent on the input data and are independent of the requested tasks are shared between tasks 602 and 603 that process the input video 601. Line 650 can share its output between multiple tasks processed by downstream components in line 650.

[0161] Encoder 312 may rely on the requested task to identify the correct annotations for the image frames of input video 601, but it may rely more on the input data and may also be shared between different tasks. Thus, line 650 may share encoder 632 between tasks 602 and 603. Furthermore, sharing encoder 632 across tasks may improve its training due to the greater number of samples available across multiple tasks.

[0162] The quality network 315 acts directly on the quality of the image, and is not dependent on the requested task. Therefore, using a separate instance of the quality network 315, since the quality score of the image frame in the input video (e.g., input video 601) has no relationship with the requested task (e.g., tasks 602 and 603), each task becomes redundant and will result in the same operation being applied multiple times on the input video 601.

[0163] Compared to the above components, segmentation network 316 is more dependent on the requested task. However, it can still be shared because it is easier to generate multiple outputs for different tasks (e.g., tasks 602 and 603). As shown in Figure 6B, segmentation network 636 is a modified version of segmentation network 316, which can return multiple segmentation masks for each task for each image frame as matrix 653.

[0164] The neural networks 633-634 may include any instance of the RNN 313 or TCN 314 that generates matrices of feature scores specific to identifying requested features of interest in different tasks. The local spatiotemporal processing module 630 of circuit 650 may be configured to generate multiple copies of the encoder outputs 637 and 638 and provide them as inputs to multiple neural networks 633 and 634, one for each task.

[0165] Figure 6C and Fig.6D An example circuit arrangement for performing multiple tasks with aggregate outputs using an example smart detector system is shown consistent with an embodiment of the present disclosure. Figure 6CAs shown, lines 610 and 620 generate outputs by using multiple time series analysis modules 671-673 simultaneously. For example, time series analysis module 673 treats input data generated by both lines 610 and 620 as input data. Time series analysis modules 671 and 672 can generate outputs similar to those described above. Fig. 6A The output of the smart detector system is the output of lines 610 and 620. The additional time series analysis module 673 can aggregate the data generated by the local and global spatiotemporal modules 611-612 and 621-622.

[0166] Fig.6D 6 shows the sharing of circuits of local and global spatiotemporal modules 630 and 604 in addition to sharing the outputs of the modules for time series analysis. Fig.6D As shown, the time series analysis module 683 obtains both vectors 661 and 664, including scores of images generated using sampler 631 and encoder 632 with and without preprocessing the image. The time series analysis module 683 aggregates the data to produce a vector similar to Figure 6C The output of the time series analysis module 673 in.

[0167] Fig. 6E An exemplary panel is shown with output summaries of multiple tasks generated using an example intelligent detector system consistent with embodiments of the present disclosure. Panel 690 may provide a summary of a medical procedure, such as a colonoscopy performed by a healthcare professional. Panel 690 may provide information for different parts of the medical procedure and may also include scores and / or other information summarizing the healthcare professional's performance of the examination and features of interest identified, such as the number of polyps identified.

[0168] like Fig. 6EAs shown, panel 690 may include quality score summaries for different parts of the colon (right colon quality score summary 691, transverse colon quality score summary 692, and left colon quality score summary 693) and a whole colon quality score summary 694. The quality score summaries 691-694 may include time statistics for different actions, such as careful exploration, performing surgery, washing / cleaning mucosa, and quickly moving or navigating through the colon or other human organs. The system 100 may determine, for example, the withdrawal time and the amount of time and / or percentage of time identified as "careful exploration" based on characteristics or factors related to the behavior of the medical professional. The intelligent detector system 100 may identify actions performed by the medical professional as "careful exploration" based on, for example, the time spent by the medical professional analyzing the scanned portion of the organ relative to other portions. For example, an endoscopist analyzing the mucosa may be considered a "careful exploration" action, as opposed to other actions such as cleaning / removing lesions. Time statistics may include summaries of other actions such as performing surgery, washing / cleaning a portion of an anatomical region or organ (e.g., mucosa), and quickly moving / navigating an anatomical location or organ during a medical procedure. Different medical procedures (e.g., colonoscopy, video surgery, video capsule-based scanning) may include different actions of the healthcare professional as "careful exploration." The smart detector system 100 may be configured to label the actions of the healthcare professional as "careful exploration." The quality score summary panel 690 may also include a color-coded representation of the quality of the examination for each portion of the medical procedure. For example, Fig. 6E As shown, the quality score summary panel 690 may include traffic light colored circles or icons (eg, red, orange, and green) that are highlighted to indicate the quality level of the inspection for each portion of the process.

[0169] Figure 7 is a flow chart depicting the operation of an example method for detecting pathology in an input video of an image consistent with embodiments of the present disclosure. The steps of method 700 may be performed by Figure 1A The smart detector system 100 uses Figure 2 It should be understood that the illustrated method 700 may be altered to modify the order of steps and include additional steps.

[0170] In step 710, the smart detector system 100 may receive an input video or ordered set of images via the network 160. As disclosed herein, the images to be processed may be temporally ordered. The smart detector system 100 may request images directly from the image source 150. In some embodiments, other external devices such as the physician device 180 and the user device 170 may direct the smart detector system 100 to request the image source 150 to acquire images. In some embodiments, the user device 170 may submit a request to detect a feature of interest in an image currently being streamed or otherwise received by the image source 150.

[0171] In step 720, the smart detector system 100 may analyze the image subsets individually to determine characteristics associated with each requested feature of interest. The smart detector system 100 may use the sampler 311 (e.g., Figure 1A As shown) using the local spatiotemporal processing module 110 (as Figure 1A ) to select a subset of images for analysis. In addition, as disclosed herein, the local spatiotemporal processing module 110 may have a limited subset of images when determining features in an image.

[0172] The smart detector system 100 may allow for configuration of the number of images to include in an image subset, as disclosed herein. The smart detector system 110 may automatically configure the size of the subset based on the requested feature of interest or characteristics associated therewith. In some embodiments, a user of the smart detector system 100 may configure the size of the subset based on a request from a user or physician (e.g., via Figure 1A The size of the subset may be configured by input from a user device 170 or a physician device 180). The image subsets may overlap and share images between them. The smart detector system 100 may allow configuration of the number of overlapped images between the image subsets processed by the local spatiotemporal processing module 110. The smart detector system 100 may select subsets of images simultaneously. In some embodiments, the smart detector system 100 may receive a stream of images from an image source 150 and may store them in a buffer until the number of images required to form a subset is achieved.

[0173] The smart detector system 100 can use the local spatiotemporal processing module 110 to analyze the image subset to determine the likelihood of a feature in each image of the image subset. The likelihood of a feature associated with each feature of interest can be represented by a range of continuous or discrete values. For example, a value ranging between 0 and 1 can be used to represent the likelihood of a feature.

[0174] The smart detector system 100 may detect the characteristic by encoding each image of the subset of images using the encoder 312. As part of the analysis process, the smart detector system 100 may use a recurrent neural network (e.g., Figure 3AIn some embodiments, the smart detector system 100 may use a causal temporal convolutional network (e.g., Figure 3A The TCN 314 shown is used to extract spatiotemporal information of the characteristics determined in each image of the image subset.

[0175] The smart detector system 100 can use the quality network 315 information (such as Figure 3A The smart detector system 100 may use the quality network 315 to determine a vector of quality scores corresponding to each image of the subset of images. The quality scores may be used to rank each image relative to an ideal image having desired characteristics. The quality network 315 may output the quality scores as sequence numbers. The sequence numbers may be numeric ranges where the image quality is too poor and needs to be ignored. For example, the quality network 315 may output a quality score between 0 and R.

[0176] In some embodiments, the smart detector system 100 can generate additional information about the feature using the segmentation network 316. The additional information can include information about the image portion in each image. The smart detector system 100 can use the segmentation network 316 to extract the portion of the image having the requested feature of interest by generating a segmentation mask for each image of the image subset. The segmentation network 316 can use a deep convolutional neural network to extract the image.

[0177] In step 730, the smart detector system 100 may process the vector of information about the images and the image characteristics determined in step 720. The smart detector system 100 may use the global spatiotemporal processing module 120 to process the output generated by the local spatiotemporal processing module 110 in step 720. The smart detector system 100 may process the information vectors associated with all images together to refine the information vectors, including the characteristics determined in each image. The global spatiotemporal processing module 120 may apply a non-causal temporal convolutional network (e.g., Figure 3B The temporal convolutional network 321) is used to refine the feature information generated by the components of the local spatiotemporal processing module 110.

[0178] The smart detector system 100 may also use a post-processor (e.g., Figure 3B The post-processor 322 shown refines the vector with additional information about the image and features (e.g., quality scores and segmentation masks). The smart detector system 100 can use one or more signal processing techniques to refine the quality score of each image of the ordered set of images. By way of example, the smart detector system 100 can use one or more signal processing techniques such as low-pass filtering or Gaussian smoothing to refine the quality score.

[0179] For example, Figure 5A As shown, post-processor 523 may take quality score matrix 532 of quality scores to generate refinement score matrix 542 .

[0180] In some embodiments, the smart detector system 100 may use a post-processor (e.g., Figure 3B The post-processor 322 shown in FIG. 1 is used to refine the segmentation mask for image segmentation to extract the portion of each image containing the requested feature of interest. The smart detector system 100 can use morphological operations to refine the segmentation mask by utilizing prior information about the shape and distribution of the feature or features of interest in the ordered image collection. For example, Figure 5A As shown, the post-processor 524 may take the matrix of the segmentation mask 533 as input to generate a refinement matrix of the segmentation mask 543 .

[0181] In step 740, the smart detector system 100 may associate a numerical value with each image based on the refined characteristics for each image of the ordered set of images in step 730. Components of the smart detector system 100 may interpret the assigned numerical value for each image to determine a probability of identifying a feature of interest within each image. The smart detector system 100 may present different numerical values ​​to indicate different states for each requested feature of interest. For example, the smart detector system 100 may output a first numerical value for each image in which a requested feature of interest is detected, and output a second numerical value for each image in which a requested feature of interest is not detected.

[0182] In some embodiments, the smart detector system 100 may interpret the associated numerical values ​​to determine the location of the requested feature of interest in the image or the number of images that include the feature. After step 740, the smart detector system 100 may generate a report with information about each feature of interest based on the numerical values ​​associated with each image (step 750). As described above, the report may be presented electronically in a different form (e.g., file, display, data transmission, etc.) and may include information about the presence of each requested feature of interest and additional information and / or recommendations based on, for example, medical guidelines. Once step 750 is completed, the smart detector system 100 completes execution of the process (step 799) and method 700 on, for example, a computing device 200.

[0183] Figure 8 is a flow chart depicting the operation of an example method for spatiotemporal analysis of video content consistent with embodiments of the present disclosure. The steps of method 800 may be performed by Figure 1A The smart detector system 100 uses Figure 2It should be understood that the illustrated method 800 may be altered to modify the order of steps and include additional steps.

[0184] In step 810, the smart detector system 100 can be connected to the network 160 (eg Figure 1A ) from an image source 150 (as shown Figure 1A In some embodiments, the smart detector system 100 can access images by extracting images from the input video. In some embodiments, the received images can be stored in a memory and accessed from the memory.

[0185] In step 820, the smart detector system 100 may use the spatiotemporal information of the features in each image of the ordered image set to detect the occurrence of an event in the temporally ordered image set. The smart detector system 100 may use the event detector 331 (e.g., Figure 3C The smart detector system 100 may use the local spatiotemporal processing module 110 and the global spatiotemporal processing module 120 to determine the spatiotemporal information. The smart detector system 100 may determine the spatiotemporal information in a two-step manner. First, the local spatiotemporal processing module may retrieve the spatiotemporal information about the characteristic by examining each image of the accessed image collection. In some embodiments, the local spatiotemporal processing module 110 may use a subset of the images. Second, the global spatiotemporal processing module 120 may use the spatiotemporal information about the local characteristic of each image to generate combined spatiotemporal information of all images by examining the spatiotemporal information of all images generated by the local spatiotemporal processing module 110.

[0186] Upon detecting an event, the smart detector system 100 may add color to a portion of the timeline of the video content that matches a subset of the temporally ordered set of images of the video content in which the event was found.

[0187] The color may vary with the level of correlation of the images of a subset of the temporally ordered set of images for a characteristic related to the feature of interest. The color may vary with the level of correlation of the images of a subset of the temporally ordered set of images for one or more characteristics.

[0188] The smart detector system 100 may use the determined spatiotemporal information of the characteristics to determine the presence of an event representing the occurrence of a feature of interest in the temporally ordered set of images.

[0189] In step 830, the smart detector system 100 may use the frame selector 332 (e.g., based on the correlation score and the quality score of the image indicating the presence of a characteristic associated with the at least one feature of interest) to select the image. Figure 2) selects an image from the image collection. The smart detector system 100 may use the quality network 335 to evaluate the quality score of each image of the images accessed in step 810. The frame selector 332 may examine the images and determine images with information using the quality scores generated by the quality network 335 and the characteristic scores generated in step 820. The smart detector system 100 may select image frames by adding bookmarks to the images in the temporally ordered image collection.

[0190] In step 840, the smart detector system 100 may merge subsets of images with matching features based on spatial and temporal correlation using the object descriptor 333. The smart detector system may use the spatiotemporal information of the features in each image determined in step 820 to determine the spatiotemporal correlation of the features.

[0191] In step 850, the smart detector system 100 may use the time slicer 334 (e.g. Figure 3C The smart detector system 100 can segment a temporally ordered set of images that satisfy the temporal correlation of the selected task by identifying a subset of the image set where one or more features of interest exist. The smart detector system 100 can use the spatiotemporal information of the features determined in step 820 to determine the temporal correlation. If a match exists between an image and one or more features of interest, the smart detector system 100 can consider the image to be temporally correlated.

[0192] The smart detector system 100 may extract a segment of the video content that matches one of the segmented subsets of the temporally ordered set of images of the video. The extracted segment may include at least one feature of interest. Once step 850 is completed, the smart detector system 100 completes execution of the process (step 899) and method 800 on the computing device 200.

[0193] Fig. 9 is a flow chart describing the operation of an exemplary method 900 for performing multiple tasks on a set of input images consistent with an embodiment of the present disclosure. The steps of method 900 may be performed by Figure 1A The smart detector system 100 uses Figure 2 It should be understood that the illustrated method 900 may be altered to modify the order of steps and include additional steps.

[0194] In step 910, the smart detector system 100 may receive a plurality of tasks (eg, Fig. 6A 602 and 603) and an input video comprising a set of images (e.g., Fig. 6AEach of the received tasks 602 and 603 may include a request to identify a feature of interest in a set of input images in the input video.

[0195] In step 920, the smart detector system 100 may use the local spatiotemporal processing module 110 (eg, Figure 1A ) analyzes a subset of images to identify the presence of characteristics associated with each requested feature of interest in each image of the subset of images.

[0196] In some embodiments, the smart detector system 100 may use the global spatiotemporal analysis module 120 to refine the features identified by the local spatiotemporal processing module 110 by filtering incorrectly identified features. In some embodiments, the global spatiotemporal processing module 120 may highlight and mark some features identified by the local spatiotemporal processing module 110. In some embodiments, the global spatiotemporal processing module 120 may filter the image collection using additional components such as the quality network 315 and the segmentation network 316 to generate additional information about the input images.

[0197] In step 930, the smart detector system 100 may iteratively execute the time series analysis module 130 for each task of the requested task set to associate a numerical score with each image of the input image set. In some embodiments, the smart detector system 100 may include multiple instances of the time series module 130 to process multiple tasks simultaneously. For example, the time series modules 671 and 672 (e.g., Figure 6B ) simultaneously identifies different sets of features in the same set of images for different tasks 602 and 603. Once step 930 is completed, the smart detector system 100 completes execution of the process (step 999) and method 900 on the computing device 200.

[0198] The figures and components in the accompanying drawings described above illustrate the possible implementation architecture, functions and operations of the system, method and computer hardware or software products according to various exemplary embodiments of the present disclosure. For example, each block in the flowchart or figure may represent a module, segment or portion of a code, which includes one or more executable instructions for implementing a specified logical function. It should also be understood that in some alternative embodiments, the functions indicated in the block may not occur in the order indicated in the accompanying drawings. By way of example, two blocks or steps shown in succession may be executed or implemented substantially simultaneously, or two blocks or steps may sometimes be executed in reverse order according to the functions involved. In addition, some blocks or steps may be omitted. It should also be understood that each block or step of the figure and the combination of blocks or steps may be implemented by a dedicated hardware-based system that performs a specified function or action, or by a combination of dedicated hardware and computer instructions. Computer program products (e.g., software or program instructions) may also be implemented based on the described embodiments and the examples shown.

[0199] It should be understood that the above systems and methods can be varied in many ways, and different features can be combined in different ways. Specifically, not all of the features shown above in a particular embodiment or implementation are required in every embodiment or implementation. Further combinations of the above features and implementations are also considered to be within the scope of the embodiments or implementations disclosed herein.

[0200] Although certain embodiments and features of the implementation have been described and illustrated herein, modifications, substitutions, changes, and equivalents will be apparent to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and variations that fall within the scope of the disclosed embodiments and the features of the illustrated implementations. It should also be understood that the embodiments described herein are presented by way of example only and not limitation, and that various changes may be made in form and detail. Any part of the system and / or method described herein may be implemented in any combination except mutually exclusive combinations. By way of example, the implementations described herein may include various combinations and / or sub-combinations of the functions, components, and / or features of the different embodiments described.

[0201] In addition, although illustrative embodiments have been described herein, the scope of the present disclosure includes embodiments having equivalent elements, modifications, omissions, combinations (e.g., aspects across multiple embodiments), adaptations, or changes based on the embodiments disclosed herein. In addition, the elements in the claims will be broadly interpreted based on the language used in the claims and are not limited to the examples described herein or during the prosecution of this application. On the contrary, these examples will be interpreted as non-exclusive. Therefore, this specification and examples are considered to be exemplary only, and their true scope and spirit are indicated by the full scope of the attached claims and their equivalents.

Claims

1. A computer-implemented system comprising: one or more memory devices storing processor-executable instructions; and One or more processors configured to execute the instructions to cause the system to perform operations to perform spatiotemporal analysis of images captured with an imaging device, the operations comprising: accessing a temporally ordered set of images from the captured images; detecting an occurrence of an event in the temporally ordered set of images using an event detector, wherein a start time and an end time of the event are identified by a start image frame and an end image frame in the temporally ordered set of images; selecting, using a frame selector, a group of images from a set of images in the temporally ordered set of images bounded by the start image frame and the end image frame based on relevance scores and quality scores of the images, wherein the relevance scores of the selected images indicate the presence of at least one feature of interest; merging a subset of images from the selected images based on matching presence of the at least one feature of interest using an object descriptor, wherein the subset of images is identified based on spatial and temporal correlation using spatiotemporal information; and The temporally ordered set of images is segmented using a temporal segmenter at time intervals satisfying the temporal relevance of a selected task.

2. The system according to claim 1, wherein: The one or more processors are further configured to: Determining, using a local spatiotemporal processing module, spatiotemporal information of characteristics associated with the at least one feature of interest of a subset of images of the video content; and A global spatiotemporal processing module is used to determine spatiotemporal information of all images of the video content.

3. The system according to claim 1 or 2, wherein: In order to segment the temporally ordered set of images by time intervals, the one or more processors are further configured to: identifying a subset of said temporally ordered set of images having the presence of said at least one feature of interest; or A subset of the temporally ordered set of images having an event present is identified.

4. The system according to claim 3, wherein: To identify a subset of the temporally ordered set of images having the presence of the at least one feature of interest, the one or more processors are further configured to: A bookmark is added to an image in the temporally ordered set of images, wherein the bookmarked image is part of a subset in the temporally ordered set of images.

5. The system according to claim 3 or 4, wherein: To identify a subset of the temporally ordered set of images having the presence of the at least one feature of interest, the one or more processors are further configured to: A set of images is extracted from a subset of the temporally ordered set of images.

6. The system according to claim 5, wherein: The extracted set of images includes characteristics associated with the at least one feature of interest.

7. A system according to any one of claims 3 to 6, wherein: To identify a subset of the temporally ordered set of images having the presence of the at least one feature of interest, the one or more processors are further configured to: Color is added to a portion of the timeline of captured images that matches a subset of the temporally ordered set of images.

8. The system according to claim 7, wherein: The color varies with a correlation level of images of a subset of the temporally ordered set of images of the at least one feature of interest.

9. The system according to claim 7 or 8, wherein: The color varies with a level of correlation of images of a subset of the temporally ordered set of images of a characteristic associated with the at least one feature of interest.

10. The system according to any one of claims 7 to 9, wherein: The color varies with different features of interest associated with the at least one feature of interest.

11. A system according to any one of claims 7 to 10, wherein: The color varies with different events detected using the event detector.

12. A system according to any one of claims 7 to 11, wherein: The timeline is presented as part of a video summary, wherein the video summary includes overlaid text and graphics.

13. The system according to claim 12, wherein: The video summary is generated by selecting relevant frames from the captured images and has a variable frame rate video output.

14. A system according to any one of the preceding claims, wherein: The occurrence of the event represents a part of a medical procedure.

15. A system according to any one of the preceding claims, wherein: The operations also include: A panel having a summary of the temporally ordered set of images is generated, wherein the summary includes images selected using the frame selector and enhanced with display tags.

16. The system of claim 15, wherein: The generated panel includes a quality score of the medical procedure performed when the image was captured using the imaging device.

17. The system according to claim 15 or 16, wherein: The generated panel includes a quality score for an operator of the imaging device performing the medical procedure.

18. A system according to any one of claims 15 to 17, wherein: The generated panel includes aggregated information from one or more of the event detector, the frame selector, the object descriptor, and the time slicer.

19. A system according to any one of the preceding claims, wherein: The ordered set of images is received directly from the imaging device during a medical procedure.

20. A system according to any one of the preceding claims, wherein: The presence of the at least one feature of interest is determined based on a portion of the captured image.

21. A system according to any one of the preceding claims, wherein: The system is configured to perform operations to perform a plurality of tasks on a collection of images, the operations comprising: receiving a plurality of tasks, wherein at least one task of the plurality of tasks is associated with a request to identify at least one feature of interest in the collection of images; analyzing a subset of images of the image collection using a local spatiotemporal processing module to identify the presence of a characteristic associated with the at least one feature of interest; and The time series analysis module is iteratively executed for each task in the plurality of tasks to associate a numerical score for each task with each image of the subset of images.

22. The system of claim 21, wherein: The local spatiotemporal processing module outputs subsets of analyzed images of the set of images, wherein each subset is associated with a task of the plurality of tasks.

23. The system according to claim 21 or 22, wherein: The local spatiotemporal processing module determines the presence of the characteristic by determining a vector of quality scores, wherein each quality score in the vector of quality scores corresponds to each image of the subset of images.

24. A system according to any one of claims 21 to 23, wherein: The local spatiotemporal processing module generates a set of feature vectors for features of interest associated with the plurality of tasks.

25. A system according to any one of claims 21 to 24, wherein: The operations also include: The set of feature vectors of the subset of images analyzed by the local spatiotemporal processing module is analyzed using a global spatiotemporal processing module.

26. A system according to any one of claims 21 to 25, wherein: The operations also include: The output of the local spatiotemporal processing module is aggregated for each of the plurality of tasks using the time series analysis module.

27. A computer-implemented method for performing spatiotemporal analysis of images captured with an imaging device, the method comprising the following operations performed by at least one processor: accessing a temporally ordered set of images from the captured images; detecting an occurrence of an event in the temporally ordered set of images using an event detector, wherein a start time and an end time of the event are identified by a start image frame and an end image frame in the temporally ordered set of images; selecting, using a frame selector, a group of images from a set of images in the temporally ordered set of images bounded by the start image frame and the end image frame based on relevance scores and quality scores of the images, wherein the relevance scores of the selected images indicate the presence of at least one feature of interest; merging a subset of images from the selected images based on matching presence of the at least one feature of interest using an object descriptor, wherein the subset of images is identified based on spatial and temporal correlation using spatiotemporal information generated by a local spatiotemporal processing module; as well as The temporally ordered set of images is segmented using a temporal segmenter at time intervals satisfying the temporal relevance of a selected task.

28. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations for performing spatiotemporal analysis of images captured with an imaging device, the operations comprising: accessing a temporally ordered set of images from the captured images; detecting an occurrence of an event in the temporally ordered set of images using an event detector, wherein a start time and an end time of the event are identified by a start image frame and an end image frame in the temporally ordered set of images; selecting, using a frame selector, a group of images from a set of images in the temporally ordered set of images bounded by the start image frame and the end image frame based on relevance scores and quality scores of the images, wherein the relevance scores of the selected images indicate the presence of at least one feature of interest; merging a subset of images from the selected images based on matching presence of the at least one feature of interest using an object descriptor, wherein the subset of images is identified based on spatial and temporal correlation using spatiotemporal information generated by a local spatiotemporal processing module; as well as The temporally ordered set of images is segmented using a temporal segmenter at time intervals satisfying the temporal relevance of a selected task.

Images