Method for ultrasound image processing and apparatus for implementing the method - Patent Application 20070122997

Abstract

A method is proposed for processing a set of ultrasound images generated for observing a target organ, the method comprising the steps of: determining, for a result image of the set of ultrasound images, using a first machine learning algorithm, a prediction of a class of the result image, the class belonging to a set of classes including at least one class related to the target organ; determining whether the class is related to the target organ; and, based on the determination that the class is related to the target organ, generating an output signal using a prediction of detection of the presence in the result image of a structure related to a procedure for observing the target organ determined based on the set of ultrasound images.

Description

[Technical Field]

[0001] The present disclosure relates to the field of image processing, particularly for ultrasound video stream processing. [Background technology]

[0002] Endoscopic ultrasound (EUS) and magnetic resonance imaging (MRI) are considered today the most precise tools for organ imaging (e.g., pancreatic imaging). Generally, EUS is considered superior for detecting small lesions (tumors of T1 or less). Despite its strong potential, the general lack of access to high-quality EUS has limited its usefulness. Indeed, EUS is extremely difficult to master, requiring advanced technical skills as well as a level of training and extensive experience that significantly exceeds that of typical endoscopy. The difficulty of performing a high-quality EUS procedure is related to two main issues: the difficulty in understanding the orientation and position of the scope tip within the patient (technical skill) and the correct interpretation of the ultrasound images (interpretation skill).

[0003] Artificial intelligence (AI), and especially deep learning (DL), has the potential to improve EUS practice by assisting physicians in identifying structures in ultrasound images. DL algorithms, especially convolutional neural networks (CNNs), are a key asset for structure recognition because they can be trained to learn and then identify relevant features from images without explicit human intervention.

[0004] Nevertheless, no scheme has yet been proposed to provide improved EUS procedures by combining them with AI-based techniques.

[0005] Therefore, there is a need for an improved ultrasound video processing scheme and apparatus for implementing the scheme that addresses at least some of the above-described drawbacks and shortcomings of the prior art. Summary of the Invention [Problem to be solved by the invention]

[0006] It is an object of the present disclosure to provide an improved ultrasound video processing scheme and apparatus for implementing the scheme.

[0007] Another object of the present disclosure is to provide an improved scheme for processing a set of ultrasound images and an apparatus for implementing the scheme.

[0008] Another object of the present disclosure is to provide an improved method for processing a set of ultrasound images generated for observing a target organ, for example, by an ultrasound endoscope, and an apparatus for implementing the method, in order to mitigate the above-mentioned drawbacks and shortcomings of conventional video coding schemes. [Means for solving the problem]

[0009] To achieve these objectives and other advantages, and in accordance with the objectives of the present disclosure as embodied and broadly described herein, one aspect of the present disclosure proposes a method for processing a set of ultrasound images, e.g., generated by an ultrasound endoscope for observing a target organ. The proposed method includes the steps of: determining, for a result image of the set of ultrasound images, using a first machine learning algorithm, a prediction of a class of the result image, the class belonging to a set of classes including at least one class related to the target organ; determining whether the class is related to the target organ; and, based on the determination that the class is related to the target organ, generating an output signal, e.g., including one or more of an audio signal and an augmented result image, using the prediction of the presence of a structure related to the procedure for observing the target organ in the result image determined based on the set of ultrasound images.

[0010] In some embodiments, the proposed scheme enables the use of machine learning algorithms to determine a prediction of a class of result images in a set of ultrasound images and to determine whether the class is associated with a target organ prior to using another prediction to generate an output signal such as an augmented result image.

[0011] Therefore, advantageously, the determination of the predicted class of the result image may be used to first confirm that the result image is related to the target organ, and then go further and generate an output signal, such as an augmented result image, using the predicted detection of the presence in the result image of structures related to the procedure for observing the target organ determined based on the set of ultrasound images. In particular, the proposed scheme makes it possible to avoid using the predicted detection of the presence of structures in the result image to generate an output signal, such as an augmented result image, in cases where the result image is not actually related to the target organ.

[0012] Another advantage is that the proposed method can be used to process ultrasound images generated by an ultrasound endoscope to observe target organs, i.e., in the context of EUS imaging. In such a context, the proposed method may lead to significant improvements in the detection of pancreatic cancer, since one of the difficulties in using EUS imaging to observe the pancreatic organ, in addition to the inherent difficulty of mastering EUS as discussed above, relates to the need to detect small lesions and ensure that detected lesions are actually formed on the pancreatic organ.

[0013] In one or more embodiments, the set of ultrasound images may be generated by an ultrasound endoscope, although the present disclosure is not limited to ultrasound images generated by endoscopy and may apply to sets of ultrasound images generated by any suitable imaging technique.

[0014] In one or more embodiments, detecting the presence in the result image of a structure related to a procedure for observing a target organ may include detecting the presence of tissue and / or lesions of the target organ in the result image. For example, in some embodiments, detecting the presence of tissue may include an identified region in the result image (e.g., a box with a square or rectangular shape), a detection class associated with the region (e.g., presence of pancreas), and possibly a detection score (e.g., a likelihood value) measuring a confidence level, and detecting the presence of lesion may include an identified region in the result image (e.g., a box with a square or rectangular shape), a detection class associated with the region (e.g., presence of lesion), and possibly a detection score (e.g., a likelihood value) measuring a confidence level.

[0015] In one or more embodiments, the prediction of detection may be determined by a first machine learning algorithm or by a second machine learning algorithm.

[0016] In one or more embodiments, the prediction of detection may be determined by a second machine learning algorithm, and the first and second machine learning algorithms may be executed in parallel to generate the class prediction and the detection prediction based on the set of ultrasound images. Executing the first and second machine learning algorithms advantageously speeds up the completion of the first and second machine learning algorithms.

[0017] In one or more embodiments, the first machine learning algorithm may be configured to receive pre-processed images of the set of ultrasound images as input data.

[0018] In one or more embodiments, the first machine learning algorithm may be configured to implement an artificial intelligence algorithm using a neural network.

[0019] In one or more embodiments, the prediction of detection may be determined by a second machine learning algorithm, where the second machine learning algorithm may be configured to receive preprocessed images of the set of ultrasound images as input data.

[0020] In one or more embodiments, the output signal may include one or more of an audio signal, an augmented result image, and any other suitable multimedia signal.

[0021] In one or more embodiments, the first machine learning algorithm may be a supervised learning algorithm, and the proposed method may further include performing a training phase for training the supervised learning algorithm as a classification algorithm, in which the supervised learning algorithm is trained on training data.

[0022] In one or more embodiments, the target organ is the pancreas.

[0023] In one or more embodiments, an augmented result image may be generated by overlaying result image information related to the presence of tissue in the target organ and / or the presence of a lesion in the target organ based on a prediction of the detection of the presence of tissue and / or a lesion in the target organ in the result image.

[0024] In one or more embodiments, a set of ultrasound images may be extracted from an ultrasound video stream generated by an ultrasound endoscope to observe a target organ, and the proposed method may further include displaying the augmented ultrasound video stream including the augmented resultant images on a display.

[0025] In one or more embodiments, the prediction of the class may include a likelihood value associated with the class, and determining whether the class is associated with the target organ may be based on a comparison of the likelihood value to a predefined threshold.

[0026] In another aspect of the present disclosure, an apparatus is proposed, comprising a processor and a memory operatively coupled to the processor, the apparatus configured to perform the method proposed in the present disclosure.

[0027] In yet another aspect of the present disclosure, a non-transitory computer-readable medium is proposed that is encoded with executable instructions that, when executed, cause an apparatus comprising a processor operatively coupled to a memory to perform the methods proposed in the present disclosure.

[0028] For example, in some embodiments, the present disclosure provides a non-transitory computer-readable medium encoded with executable instructions that, when executed, cause an apparatus having a processor operatively coupled to a memory to process a set of ultrasound images generated for observing a target organ by performing a method for processing a set of ultrasound images, the method including the steps of: determining, by the processor, for a result image of the set of ultrasound images, using a first machine learning algorithm, a prediction of a class of the result image, the class belonging to a set of classes including at least one class associated with the target organ; determining, by the processor, whether the class is associated with the target organ; and, based on the determination that the class is associated with the target organ, generating an output signal, such as, for example, an augmented result image, using the prediction of detection of the presence in the result image of a structure related to the procedure for observing the target organ determined based on the set of ultrasound images.

[0029] In yet another aspect of the present disclosure, there is proposed a computer program product comprising computer program code tangibly embodied in a computer readable medium, said computer program code comprising instructions that, when provided to a computer system and executed, cause said computer to perform the method proposed in the present disclosure.

[0030] In another aspect of the present disclosure, a data set is proposed that represents the computer programs proposed herein, for example through compression or encoding.

[0031] It should be appreciated that the present invention can be implemented and used in numerous ways, including, but not limited to, as a process, an apparatus, a system, a device, and a method for applications now known and later developed. These and other unique features of the systems disclosed herein will become more readily apparent from the following description and accompanying drawings.

[0032] The present disclosure may be better understood, and its numerous objects and advantages made more apparent to those skilled in the art by reference to the following drawings, taken in conjunction with the accompanying specification. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 illustrates an exemplary ultrasound image processing method in accordance with one or more embodiments of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating an exemplary imaging system, in accordance with one or more embodiments. [Figure 3a] FIG. 1 illustrates an exemplary ultrasound imaging subsystem in accordance with one or more embodiments of the present disclosure. [Figure 3b] FIG. 1 illustrates an exemplary ultrasound imaging subsystem in accordance with one or more embodiments of the present disclosure. [Figure 4] FIG. 1 illustrates an example device or unit configured to use one or more features according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0034] For brevity and clarity of explanation, the drawings show general constructions, and descriptions and details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the discussion of the described embodiments of the present invention. Additionally, elements in the drawings are not necessarily drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help enhance understanding of embodiments of the present invention. Some figures may be idealized to aid understanding, such as when structures are shown with straight lines, sharp angles, and / or parallel planes that would not be highly symmetrical or regular under real-world conditions. The same reference numbers in different figures may, but do not necessarily, refer to the same elements, and similar reference numbers may, but do not necessarily, refer to similar elements.

[0035] Furthermore, it will be apparent that the teachings herein may be embodied in a wide variety of forms and that any specific structure and / or function disclosed herein is merely representative. In particular, those skilled in the art will understand that aspects disclosed herein can be implemented independently of any other aspect, and that certain aspects can be combined in various ways.

[0036] The present disclosure is described below with reference to functions, engines, block diagrams, and flowchart illustrations of methods, systems, and computer programs according to one or more exemplary embodiments. Each described function, engine, or block of the block diagrams and flowchart illustrations may be implemented in hardware, software, firmware, middleware, microcode, or any suitable combination thereof. If implemented in software, the functions, engines, or blocks of the block diagrams and / or flowchart illustrations may be implemented by computer program instructions or software code, which may be stored or transmitted over a computer-readable medium, or loaded into a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program instructions or software code executing on the computer or other programmable data processing apparatus create means for implementing the functions described herein.

[0037] Embodiments of computer-readable media include, but are not limited to, both computer storage media and computer communication media, including any medium that facilitates transfer of a computer program from one place to another. As used herein, a "computer storage medium" may be any physical medium that can be accessed by a computer or processor. Furthermore, the terms "memory" and "computer storage medium" include any type of data storage device, such as, but not limited to, a hard drive, a flash drive or other flash memory device (e.g., a memory key, memory stick, key drive), a CD-ROM or other optical data storage device, a DVD, a magnetic disk data storage device or other magnetic data storage device, a data memory component, RAM, ROM and EEPROM memory, a memory card (smart card), a solid-state drive (SSD) memory, and any other form of media that can be used to transport, store, or remember data or data structures that can be read by a computer processor, or combinations thereof. Additionally, various forms of computer-readable media may transmit or carry instructions to a computer, such as a router, gateway, server, or any data transmission device, whether involving wired transmission (via coaxial cable, fiber optic, phone line, DSL cable, or Ethernet cable), wireless transmission (via infrared, radio, cellular, microwave), or virtualized transmission device (virtual router, virtual gateway, virtual tunnel end, virtual firewall). According to these embodiments, the instructions may comprise code in any computer programming language or computer program element, such as, but not limited to, Assembler, C, C++, Visual Basic, Hypertext Markup Language (HTML), Extensible Markup Language (XML), Hypertext Transfer Protocol (HTTP), Hypertext Preprocessor (PHP), SQL, MySQL, Java, JavaScript, JavaScript Object Notation (JSON), Python, and bash scripts.

[0038] Unless otherwise specified, it will be appreciated that throughout the following description, discussions using terms such as processing, calculating, computing, determining, etc. refer to actions or processes of a computer or computing system or similar electronic computing device that manipulate data represented as physical, e.g., electronic, quantities in the registers or memory of the computing system, or convert it into other data that is similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0039] The terms "comprise," "include," "have," and any variations thereof are intended to cover a non-exclusive inclusion, whereby a process, method, article, or apparatus that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or that are inherent to such process, method, article, or apparatus.

[0040] Moreover, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The terms "particularly," "for example," "example," and "typically" are used herein to describe examples or illustrations of non-limiting embodiments that do not necessarily correspond to preferred or advantageous embodiments in view of other possible aspects or embodiments.

[0041] As used herein, the terms "operably coupled," "coupled," "mounted," and "connected," and various variations and forms thereof, refer to couplings, connections, and mountings, which may be direct or indirect, and particularly include connections between electronic devices or portions of such devices that enable the operations and modes of operation described herein. Furthermore, the terms "connected" and "coupled" are not limited to physical or mechanical connections or couplings. For example, an operative coupling may include one or more wired connections and / or one or more wireless connections between two or more items of equipment that enable simplex and / or duplex communication links between the devices or portions of equipment. By way of another example, an operative coupling or connection may include a wired and / or wireless coupling that enables data communication between a server of the proposed system and another item of equipment of the system.

[0042] In this disclosure, a "server" or "platform" refers to any (virtualized or non-virtualized) service point or computing device or system, one or more databases, and / or data communication facilities that perform data processing operations. For example, without limitation, the term "server" or "platform" may refer to a physical processor operatively coupled to associated communication, database, and data storage facilities, or to a network, group, set, or combination of processors and associated data storage and network-connected devices, as well as an operating system and one or more database systems, and application software that supports the services and functions provided by the server. A server or platform may be configured to operate within or as part of a cloud computing environment. A computing device or system may be configured to send and receive signals, or process and / or store data or signals, via wireless and / or wired transmission networks, and thus operate as a server. Devices configured to operate as servers may therefore include, by way of non-limiting example, rack-mounted dedicated servers, cloud-based servers, desktop computers, laptop computers, service gateways (sometimes called "boxes" or "home gateways"), multimedia decoders (sometimes called "set-top boxes"), integrated devices that combine various functionalities, such as two or more of the above-mentioned functionalities. Although servers may vary greatly in their configuration or their capabilities, they generally include one or more central processing units and memory.A server may also include one or more items, such as a mass memory device, one or more power sources, one or more wireless and / or wired network interfaces, one or more input / output interfaces, one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or equivalent.

[0043] As used herein, the terms "network" and "communications network" refer to one or more data links that may couple or connect computer systems and / or modules and / or other devices or electronic equipment, such as between a server and a client device or other type of device, including between wireless devices coupled or connected via a wireless network, potentially coupling or connecting virtualization appliances to transport electronic data between the computer systems and / or modules and / or other devices or electronic equipment. A network may also include mass memory for storing data, such as a network-attached storage (NAS), a storage area network (SAN), or any other form of computer-readable or machine-readable medium. A network may include, in whole or in part, the Internet, one or more local area networks (LANs), one or more wide area networks (WANs), wired connections, wireless connections, cellular connections, or any combination of these various networks. Similarly, subnetworks may use different architectures or conform to or be compatible with different protocols to interoperate with larger networks. Different types of equipment may be used to enable the interoperability of different architectures or protocols. For example, a router may be used to provide a communication or data link between two LANs that are normally separate and independent.

[0044] Largely due to insufficient early cancer detection, pancreatic cancer is currently the sixth leading cause of cancer-related deaths in France, with a 5-year survival rate of 15% and 11,456 deaths per year reported in 2018. Conversely, early detection and treatment can result in survival rates of approximately 80%. Various strategies are being pursued for earlier pancreatic cancer detection, with advanced image-based diagnostic techniques, such as endoscopic ultrasound (EUS), showing particular promise.

[0045] EUS, a minimally invasive procedure that combines endoscopy with ultrasound to obtain detailed images of the internal organs of the chest, upper abdomen, or colon, is now considered one of the best techniques for detecting pancreatic cancer lesions smaller than 20 mm in size that are suitable for surgical resection and treatment. Despite its strong potential, EUS is difficult to master, requiring advanced technical skills as well as a level of training and extensive experience that significantly exceeds that of basic endoscopy. This has led to a shortage of well-trained clinicians to provide this potentially life-saving technique to the population. The problems relate to two main areas: spatial awareness and imaging.

[0046] Generally, objects in natural images are easily identified by humans based on their texture, color, shape, and other basic image features. Recognizing anatomical structures from EUS videos is more complex, especially for inexperienced observers. Structures such as blood vessels, bile ducts, or cysts can be easily confused, even by trained observers. Furthermore, the boundaries of some structures, especially the pancreatic parenchyma, are difficult to determine.

[0047] Despite all these challenges, deep learning (DL) models still manage to detect lesions, and pancreatic parenchymal ultrasound images generated by ultrasound endoscopes are used to observe patients' pancreases. However, the performance of AI models (such as DL models) can be ineffective in detecting objects with irregular structures, i.e., when the center of gravity is difficult to determine, which is usually the case for pancreatic parenchyma.

[0048] According to the present disclosure, machine algorithms may advantageously be used to complement the use of a DL model (e.g., CNN) trained to detect the presence of structures related to the procedure for observing the target organ in resultant images extracted from a set of ultrasound images with a classifier, i.e., an AI model configured to classify the resultant images within a class of images related to the target organ or within a class of images not related to the target organ.

[0049] In some embodiments, a machine algorithm may be used to perform preliminary post-processing of the resulting image prior to subjecting the image to processing by the DL model. The machine algorithm may be configured, for example by training, to determine a prediction of the class of the resulting image, which may be used to determine whether the resulting image is associated with the target organ.

[0050] In some embodiments, a preliminary sorting of the result images to be processed may be performed to distinguish between result images that are relevant to the target organ and those that are not.

[0051] In some embodiments, only the resultant images determined to be associated with the target organ may be subjected to further processing, including, for example, generating an augmented resultant image. The resultant images determined not to be associated with the target organ may, for example, be discarded.

[0052] The methods proposed in this disclosure may be implemented by any computer system configured for image processing using AI models, such as convolutional neural networks.

[0053] FIG. 1 illustrates an exemplary image processing method 100 according to one or more embodiments of the present disclosure.

[0054] The ultrasound images to be processed are considered as inputs of the proposed method and may hereafter be variously referred to as result images, "original" images, or "input" images. The result images may be selected from a set of ultrasound images, which may hereafter be variously referred to as an ultrasound video sequence, an "original" video sequence, or an "input" video sequence.

[0055] In one or more embodiments, a set of ultrasound images may have been generated as part of a procedure for viewing a target organ, e.g., using an ultrasound endoscope. The resulting images may be selected based on any suitable selection criteria, e.g., taking into account applicable image applications. For example, video batches of a predetermined duration may be periodically extracted from an ultrasound video stream generated by the ultrasound endoscope, and the resulting images may be selected from the extracted batches.

[0056] In one or more embodiments, a first machine learning algorithm may be used to determine a prediction of the class of the resulting image 101. The class may belong to a set of classes, including at least one class associated with the target organ.

[0057] In some embodiments, a set of classes may be defined, and a machine learning algorithm may be configured (e.g., through training) to predict, based on the input image data, one or more classes (belonging to the predefined set of classes) corresponding to the input image data.

[0058] At least one of the predefined classes can be defined as associated with a target organ. For example, the predefined set of classes can be defined to include a first subset of classes associated with the target organ and a second subset of classes not associated with the target organ.

[0059] In one or more embodiments, the data of the resulting image, possibly after some preprocessing, for example for the purpose of reformatting, may be fed as input data to a first machine learning, which may produce a class prediction as output.

[0060] In some embodiments, the resulting image may be divided into blocks and, taking into account its composition, may be subjected to any suitable pre-processing for the purpose of being fed into a machine learning algorithm.

[0061] In one or more embodiments, a determination 102 may be made as to whether the predicted class is associated with the target organ.

[0062] For example, in some embodiments, it may be determined whether the predicted class belongs to a subset of classes associated with the target organ or to a subset of classes not associated with the target organ.

[0063] In one or more embodiments, the target organ is a human organ, such as the pancreas.

[0064] In one or more embodiments, if the predicted class is determined to be associated with the target organ, a prediction of the presence detection in the resultant image of a structure related to the procedure for observing the target organ may be obtained, and the prediction of the presence detection in the resultant image may be used to generate an output signal. Depending on the embodiment, the output signal may include one or more of an audio signal, a video signal, and any other multimedia signal suitable for indicating the presence detection in the resultant signal.

[0065] In one or more embodiments, if the predicted class is determined to be associated with the target organ, a prediction of the presence detection in the result image of a structure related to the procedure for observing the target organ may be obtained, and an augmented result image may be generated using the prediction of the presence detection in the result image (103).

[0066] In one or more embodiments, a set of ultrasound images may be extracted from an ultrasound video stream generated by an ultrasound endoscope to observe a target organ, and an enhanced ultrasound video stream including the enhanced resultant images may be displayed on a display device.

[0067] In some embodiments, a prediction of the presence of target organ tissue and / or target organ lesions in the resultant image may be obtained based on a determination that a class is associated with the target organ.

[0068] In some embodiments, detecting the presence of a structure in the result image related to the procedure for observing the target organ may include detecting the presence of tissue and / or a lesion of the target organ or detecting the presence of a landmark located near the target organ in the result image. In such embodiments, based on the prediction of detecting the presence of tissue and / or a lesion of the target organ in the result image, an augmented result image may be generated by overlaying on result image information related to the presence of tissue of the target organ and / or the presence of a lesion of the target organ.

[0069] In some embodiments, the detection may be configured to relate to a target organ, which may be any type of human organ, such as, for example, a pancreas, a blood vessel, a duct (e.g., of the bile duct type), a lymph node, etc. In other embodiments, the detection may be configured to relate to the environment of the target organ, such as, for example, a landmark located in the vicinity of the target organ. In still other embodiments, the detection may be configured to relate to structures other than the target organ itself, such as, for example, an object linked to a procedure performed by a practitioner, such as, for example, a biopsy needle.

[0070] In one or more embodiments, the prediction of the detection of the presence in the result image of a structure related to a procedure for observing a target organ may be generated by a machine learning algorithm executed based on input data including data of the result image, possibly after preprocessing.

[0071] Depending on the embodiment, the detection prediction may be determined by a first machine learning algorithm or by a second machine learning algorithm. Thus, in one or more embodiments, a first machine learning algorithm configured as a classifier and a second machine learning algorithm configured as a detector may be used to implement the proposed method. In some embodiments, the first and second machine learning algorithms may be run in parallel to generate class predictions and detection predictions based on a set of ultrasound images, advantageously improving the response time of a system implementing the proposed process.

[0072] In some embodiments, the procedure for viewing the target organ may be an endoscopic ultrasound imaging procedure, another ultrasound procedure, or a magnetic resonance imaging procedure.

[0073] In some embodiments, the first machine learning algorithm is configured to receive preprocessed images of the set of ultrasound images as input data. The preprocessing performed on the images of the set of ultrasound images may include resizing, converting to tensors, normalizing, and / or rescaling one or more images.

[0074] In embodiments in which the prediction of detection is determined by a second machine learning algorithm, the second machine learning algorithm may be configured to receive preprocessed images of the set of ultrasound images as input data. Such preprocessing performed on the images of the set of ultrasound images may include resizing, converting to tensors, normalizing, and / or rescaling one or more images, depending on the embodiment.

[0075] In some embodiments, one or more of the following preprocessing operations may be used on data provided as input to the first machine learning algorithm and / or the second machine learning algorithm: A resize preprocessing operation may advantageously be used to reduce the size of the image to be processed to reduce the computational complexity of the processing; A convert to tensor preprocessing operation may advantageously be used to optimize the format of the data to be processed when the processing is implemented, at least in part, on a graphics card or using a graphics processor unit (GPU) or, depending on the embodiment, any processing unit dedicated to neural network computations (e.g., VPU, TPU, NPU, etc.); A color space normalization preprocessing operation may advantageously be used to reduce the range of values ​​used for color coding to improve processing performance by reducing the computational complexity of the processing; In some embodiments, a preprocessing operation used during the inference phase of the second machine learning algorithm may advantageously be used during the training phase of the second machine learning algorithm, in some cases.

[0076] In one or more embodiments in which detection of the presence of tissue is used, one or more classes of the set of classes may be defined as associated with the target organ. For example, one or more classes may be defined as associated with each part of the target organ and / or each region of (and / or around) the target organ and / or that part. For example, in some embodiments, the set of one or more classes may be defined based on a set of subparts of the target organ corresponding to the division of the target organ into subparts, possibly with overlap between two neighboring subparts.

[0077] For example, in some embodiments, one or more classes may be defined to correspond to parts and / or regions of a target organ, such as the top of the target organ, the body of the target organ, and the tail of the target organ, respectively. For example, in an embodiment in which a target organ such as the pancreas is viewed using endoscopic ultrasound images, the following classes may be defined: HEAD, BODY, and TAIL, or, as another example, {#1=HEAD;#2=BODY;#3=TAIL;#4=OTHER;#5:ARTIFACTS}, each associated with parts and / or regions of the pancreas.

[0078] In one or more embodiments, once a determination is made that a predicted class is associated with a target organ, a prediction of detection of the presence of tissue and / or lesions in the target organ can be determined to facilitate more detailed observation and / or exploration of the target organ.

[0079] For example, in some embodiments, detection of the presence of tissue may be configured to detect so-called "landmarks" corresponding to sites and / or regions of a target organ, e.g., regions and / or organs at (and / or around) the target organ and / or its site, corresponding to a determined predicted class.

[0080] For example, in an embodiment in which a target organ such as the pancreas is observed using endoscopic ultrasound images, and classes HEAD, BODY, and TAIL are defined, each associated with a region of the pancreas, detection of the presence of tissue may be configured to detect the following "landmarks" corresponding to the pancreas or regions or organs at (or surrounding) the pancreas, as defined for each of the above classes:

[0081] [Table 1]

[0082] Other additional landmarks may include, for example, landmarks for the main pancreatic duct, the common bile duct, and the gallbladder.

[0083] Such landmarks may be defined for the detection of the presence in the resultant image of structures related to the procedure for observing the target organ, and the prediction of the detection of the presence may provide one or more predictions of the detection of one or more of such landmarks.

[0084] FIG. 2 illustrates an exemplary imaging system in accordance with one or more embodiments of the present disclosure.

[0085] Shown in FIG. 2 is a system 10 that includes an ultrasound image acquisition subsystem 11 configured to deliver ultrasound images to a pre-processing subsystem 12 operably coupled thereto, for example, via a data network (not shown).

[0086] In one or more embodiments, the ultrasound image acquisition subsystem 11 may include an endoscopic ultrasound (EUS) system that may be configured to generate an ultrasound video stream as part of an EUS procedure performed on a patient (e.g., a human patient) to view an organ (e.g., the pancreas) of such patient.

[0087] The ultrasound images generated by subsystem 11 are provided as input data to a pre-processing subsystem 12 and to an augmented reality subsystem 13, possibly over one or more data communication networks.

[0088] For example, in some embodiments, ultrasound images may be input to the pre-processing subsystem 12 and / or the augmented reality subsystem 13 as uncompressed ultrasound video content, and the video content may be transported using a data communications link suitable for carrying raw video content, such as an SDI (Serial Digital Interface) link, a TSoIP (Transport Stream over IP) link, a Digital Visual Interface (DVI) interface link, or an HDMI (High-Definition Multimedia Interface) link, possibly using a multimedia interface splitter.

[0089] The pre-processing subsystem 12 may be configured to receive ultrasound image content (eg, to receive an ultrasound video stream) and to process such content in accordance with embodiments of the present disclosure.

[0090] The pre-processing subsystem 12 may further be configured to generate, as output data, augmented reality data for the received ultrasound image content based on such content. As shown in Figure 2, the augmented reality data generated by the pre-processing subsystem 12 may be provided to the augmented reality subsystem 13 through a suitable multimedia interface or a suitable data communication interface, possibly using a data communication network.

[0091] The augmented reality subsystem 13 may be configured to receive ultrasound image content generated by the ultrasound imaging subsystem 11 and augmented reality data generated for such content by the pre-processing subsystem 12. The augmented reality subsystem 13 may be configured to generate an augmented reality ultrasound image, or, depending on the embodiment, an augmented reality ultrasound video stream, based on the received ultrasound image content and the received augmented reality data using conventional augmented reality techniques. The augmented reality ultrasound image or augmented reality ultrasound video stream may, in some embodiments, be fed to the display subsystem 14 and thereby used, for example, to assist a practitioner performing an EUS procedure to view a patient's organs.

[0092] Further details of an exemplary embodiment of the pre-processing subsystem 12 of FIG. 2 are provided in FIGS. 3a and 3b.

[0093] 3a and 3b illustrate exemplary ultrasound imaging systems in accordance with one or more embodiments of the present disclosure. Depending on the embodiment, each of the exemplary ultrasound imaging systems shown in FIGS. 3a and 3b may be implemented in pre-processing subsystem 12 of FIG. 2.

[0094] FIG. 3a illustrates an exemplary ultrasound imaging system 12a, according to some embodiments of the present disclosure.

[0095] As shown in FIG. 3a, the ultrasound image processing system 12a may be configured to receive as input data, for example, ultrasound image content or ultrasound image data 12_1 contained in a received ultrasound video stream, through an input interface (not shown).

[0096] In some embodiments, the input data received by the ultrasound image processing system 12a may include a set of ultrasound images generated to observe a target organ, and the ultrasound image processing system 12a may be configured to process the received set of ultrasound images in accordance with embodiments of the present disclosure.

[0097] In some embodiments, one or more image processing operations may be performed by preprocessing unit 12_2 on image data received by system 12a to generate input image data, which may be input to a first artificial intelligence (AI) model 12a3_1 configured to implement a first machine learning algorithm and to a second artificial intelligence model 12a3_2 configured to implement a second machine learning algorithm. Depending on the embodiment, preprocessing unit 12_2 may be configured to perform image preprocessing operations including an image resizing operation, a conversion to tensor operation, a normalization operation, and / or a rescaling operation.

[0098] In some embodiments, one or more of the following preprocessing operations may be used on data provided as input to the first machine learning algorithm (first artificial intelligence (AI) model 12a3_1) and / or the second machine learning algorithm (second artificial intelligence model 12a3_2): A resize preprocessing operation may advantageously be used to reduce the size of the image to be processed to reduce the computational complexity of the processing. A convert to tensor preprocessing operation may advantageously be used to optimize the format of the data to be processed when the processing is implemented, at least in part, on a graphics card or using a graphics processor unit (GPU) or, depending on the embodiment, any suitable processing unit (VPU, TPU, NPU, etc.) dedicated to, for example, neural network computations. A color space normalization preprocessing operation may advantageously be used to reduce the range of values ​​used for color coding to improve processing performance by reducing the computational complexity of the processing. In some embodiments, a preprocessing operation used during the inference phase of the second machine learning algorithm may advantageously be used during the training phase of the second machine learning algorithm, in some cases.

[0099] In some embodiments, different image processing operations may be performed on image data received by system 12a to generate input image data that can be input to a first artificial intelligence model 12a3_1 on the one hand and a second artificial intelligence model 12a3_2 on the other hand.

[0100] In one or more embodiments, the first machine learning algorithm may be configured as a classifier algorithm for generating classification predictions based on ultrasound image data received as input data.

[0101] In some embodiments, the first machine learning algorithm may be configured to determine a prediction of a class of ultrasound images received in the input data among a set of predefined classes.

[0102] For example, in some embodiments, the first machine learning algorithm may be configured to determine that ultrasound images received in the input data belong to one or more of a predefined set of classes.

[0103] In one or more embodiments, the predefined set of classes may include at least one class related to a target organ. For example, a class may be defined to include images on which the target organ (at least a portion thereof) can be identified. As another example, a class may be defined to include images on which a particular portion of the target organ can be identified.

[0104] Conversely, other classes may be defined to include images on which the target organ (at least a portion thereof) cannot be identified, or to include images on which a specific portion of the target organ cannot be identified.

[0105] A class defined as corresponding to an image in which the presence of a target organ can be identified may be considered to be related to the target organ, and a class defined as corresponding to an image in which the presence of a target organ cannot be identified may not be considered to be related to the target organ.

[0106] In one or more embodiments, the first machine learning algorithm may be configured to generate a confidence level associated with the prediction (e.g., including a percentage indicating the likelihood that the prediction is correct) as a complement to the class prediction.

[0107] In one or more embodiments, the second machine learning algorithm may be configured as a detection algorithm to generate a detection prediction based on ultrasound image data received as input data.

[0108] In some embodiments, the second machine learning algorithm may be configured to determine a prediction of the detection of the presence in the ultrasound image data 12_1 of a structure related to a procedure for observing a target organ.

[0109] For example, in some embodiments, the second machine learning algorithm may be configured to determine a prediction of detection of the presence of target organ tissue and / or lesion in the resultant image in the ultrasound image data 12_1. For example, in some embodiments, the second machine learning algorithm may be configured to determine a prediction of detection of the presence of target organ tissue in the ultrasound image data 12_1 and possibly a location in the ultrasound image data 12_1 where the target organ tissue is detected as being present and / or an identification of a portion of the target organ detected as being present in the ultrasound image data 12_1, and / or a prediction of detection of the presence of target organ lesion in the ultrasound image data 12_1 and possibly a location in the ultrasound image data 12_1 where the target organ lesion is detected as being present and / or an identification of a portion of the target organ where the lesion is detected as being present in the ultrasound image data 12_1.

[0110] In one or more embodiments, the classification predictions, and depending on the embodiment, the associated confidence levels, may be provided to an AR data post-processing unit 12_7 configured to process the data output by the first AI model unit 12a3_1 and the second AI model unit 12a3_2.

[0111] In some embodiments, one or more post-processing operations are performed by post-processing unit 12a4_1 on the classification prediction data generated by first AI model unit 12a3_1 before it is provided to AR data post-processing unit 12_7. Depending on the embodiment, the post-processing operations performed on the classification predictions generated by first AI model unit 12a3_1 may include processing to maximize predictions.

[0112] For example, in some embodiments, the classification prediction data may include one or more predicted classes and a prediction score (e.g., a likelihood value) generated for each predicted class. For example, in an embodiment where the pancreas is the observed target organ, the predicted classes may include the five above-discussed classes (defined as {#1=HEAD; #2=BODY; #3=TAIL; #4=OTHER; #5:ARTIFACTS}). For example, a prediction score corresponding to a confidence level may be obtained for each predicted class (e.g., a 60% confidence level for the pancreatic head class (#1=HEAD) and a 40% confidence level for the pancreatic body (#2=BODY)), and predictions may be maximized by retaining the class with the highest prediction score for later use.

[0113] In some embodiments, the data generated by post-processing unit 12a4_1 may include label data corresponding to the class having the highest score along with its score (e.g., "BODY[50%]").

[0114] In some embodiments, one or more post-processing operations are performed by post-processing unit 12a4_2 on the detection prediction data generated by second AI model unit 12a3_2 before being provided to AR data post-processing unit 12_7.

[0115] Depending on the embodiment, post-processing operations performed on the classification predictions generated by the second AI model unit 12a3_2 may include prediction filtering based on a confidence threshold and non-maximum suppression processing.

[0116] In some embodiments, predictive filtering of detection prediction data based on a confidence threshold may include the following operations: any predicted classes with low confidence levels (e.g., confidence levels below a predetermined threshold) may be discarded. A respective threshold may be chosen for each class or for each group of classes. For example, the threshold used for the class associated with pancreatic detection may advantageously be higher than the threshold used for the class associated with lesion detection, so as to increase the likelihood of detecting all pancreatic lesions, possibly at the expense of false-positive detections.

[0117] In some embodiments, non-maximum suppression processing may include the following operations: predicted classes may be analyzed to detect redundancies, and such redundancies may be discarded. For example, if an object (e.g., a lesion) is detected multiple times, only a determined number of predicted classes with the highest prediction scores may be retained for later use. In some embodiments, predicted classes identified as redundant (e.g., corresponding to respective prediction boxes on the image with intersections higher than a predetermined threshold) may be grouped, and only a predetermined number of classes (e.g., 1) with the highest prediction scores may be retained for later use.

[0118] In some embodiments, the data generated by post-processing unit 12a4_2 may include label data as described above in connection with post-processing unit 12a4_1.

[0119] Thus, in one or more embodiments, the AI ​​model unit 12a3_1 configured for classification and the AI ​​model unit 12a3_2 configured for detection may operate in parallel based on ultrasound image data 12_1 received as input data by the system 12a, possibly after respective pre-processing operations, before feeding input data to the respective AI model units to produce classification prediction data (for the AI ​​model unit 12a3_1) and detection prediction data (for the AI ​​model unit 12a3_2), which are fed to the AR data processing unit 12_7, possibly after respective post-processing operations.

[0120] In one or more embodiments, the AR data post-processing unit 12_7 may be configured to, upon receiving the classification prediction data (possibly post-processed) including the predicted class and possibly a confidence level, determine whether the predicted class is associated with the target organ.

[0121] In some embodiments in which the first AI model unit 12a3_1 is configured to generate a confidence level associated with the predicted class, determining whether the predicted class is associated with the target organ may include comparing the confidence level to a predetermined threshold.

[0122] In some embodiments, the AR data post-processing unit 12_7 may be configured to determine, based on determining that the predicted class is associated with the target organ, that the detection prediction data generated by the second AI model unit 12a3_2 may be used for the purpose of generating augmented reality data 12_8 associated with the ultrasound image data 12_1.

[0123] For example, in some embodiments, the first AI model unit 12a3_1 may be configured to predict, based on the input data, a class associated with the ultrasound image data 12_1 between two predefined classes, one of which indicates that a target organ is present on the input image data and the other of which indicates that the target organ is not present on the input image data. In such embodiments, if the class predicted for the ultrasound image data 12_1 by the first AI model unit 12a3_1 is a class that indicates that the target organ is present on the input image data, the detection prediction generated by the second AI model unit 12a3_2 may be considered confirmed regarding the presence of the target organ in the ultrasound image data 12_1, and such detection prediction may be used to generate AR data 12_8 to be used to generate augmented reality ultrasound image data. Otherwise, if the class predicted for the ultrasound image data 12_1 by the first AI model unit 12a3_1 is a class indicating that the target organ is not present on the input image data, the detection prediction generated by the second AI model unit 12a3_2 may be considered invalid regarding the presence of the target organ in the ultrasound image data 12_1, and such detection prediction should not be used to generate the AR data 12_8 to be used for generating the augmented reality ultrasound image data. Thus, the performance of the first AI model unit 12a3_1 configured to perform classification on the input ultrasound image data may advantageously be leveraged to validate or invalidate the prediction generated by the second AI model unit 12a3_2 configured to perform detection on the input ultrasound image data.

[0124] FIG. 3b illustrates another exemplary ultrasound imaging system 12b, according to some embodiments of the present disclosure.

[0125] As shown in FIG. 3b, the ultrasound image processing system 12b may also be configured to receive as input data, for example, ultrasound image content or ultrasound image data 12_1 contained in a received ultrasound video stream, through an input interface (not shown).

[0126] In some embodiments, the input data received by the ultrasound image processing system 12b may include a set of ultrasound images generated to observe a target organ, and the ultrasound image processing system 12b may be configured to process the received set of ultrasound images to generate as output data AR data 12_8, which in some embodiments may correspond to that described in connection with FIG. 3a, in accordance with an embodiment of the present disclosure.

[0127] For the purpose of processing a received set of ultrasound images according to an embodiment of the present disclosure, the ultrasound image processing system 12b shown in FIG. 3b includes a pre-processing unit 12_2, and an AR data post-processing unit 12_7, which in some embodiments may correspond to that described in relation to FIG. 3a.

[0128] In contrast to the system of FIG. 3a, the ultrasound image processing system 12b shown in FIG. 3b includes a single AI model unit 12b3 that is configured to generate both detection prediction data and classification prediction data, which in some embodiments may correspond to the detection prediction data generated by AI model unit 12a3_2 of FIG. 3a and / or the classification prediction data generated by AI model unit 12a3_1 of FIG. 3a.

[0129] In some embodiments, the prediction data generated by AI model unit 12b3 may be processed by post-processing unit 12b4, which may be configured to perform post-processing on the prediction data, such as processing to maximize classification prediction data, predictive filtering of detection prediction data based on a confidence threshold, and / or non-maximum suppression processing of detection prediction data, depending on the embodiment.

[0130] For example, in some embodiments, processing for maximizing classification prediction data may include the following operations: The classification prediction data may include one or more predicted classes and a prediction score (e.g., a likelihood value) generated for each predicted class. A prediction score for each of the predicted classes (e.g., a 60% confidence level for the pancreatic head class and a 40% confidence level for the pancreatic body class) may be obtained, and the classification prediction data may be maximized by retaining the class with the highest prediction score for later use.

[0131] In some embodiments, predictive filtering of detection prediction data based on a confidence threshold may include the following operations: any predicted classes with low confidence levels (e.g., confidence levels below a predetermined threshold) may be discarded. A respective threshold may be chosen for each class or for each group of classes. For example, the threshold used for the class associated with pancreatic detection may advantageously be higher than the threshold used for the class associated with lesion detection, so as to increase the likelihood of detecting all pancreatic lesions, possibly at the expense of false-positive detections.

[0132] In some embodiments, non-maximum suppression processing may include the following operations: predicted classes may be analyzed to detect redundancies, and such redundancies may be discarded. For example, if an object (e.g., a lesion) is detected multiple times, only a determined number of predicted classes with the highest prediction scores may be retained for later use. In some embodiments, predicted classes identified as redundant (e.g., corresponding to respective prediction boxes on the image with intersections higher than a predetermined threshold) may be grouped, and only a predetermined number of classes (e.g., 1) with the highest prediction scores may be retained for later use.

[0133] The architecture of system 12b advantageously avoids the use of two separate AI model units, even when running in parallel, providing a gain in processing time that can be leveraged to achieve higher performance for ultrasound image processing operating in real time or near real time.

[0134] The proposed method may be implemented by any image or video processing system configured to process images (or frames) of input image or video data using an AI model, such as a deep learning model implemented by a machine learning algorithm, e.g., an image or video processing system configured for processing using a deep learning model that uses a neural network such as a convolutional neural network, or an image or video processing system configured for classification using a deep learning model that uses one or more autoencoders, as the case may be, adapted to implement one or more embodiments of the proposed method.

[0135] In one or more embodiments, training of a first ML algorithm used in an image processing system of the present disclosure may be performed during a training phase of the ML algorithm, which may be designed to configure the ML algorithm (e.g., determine neural network weights) as a classifier given predefined classes, so that during an inference phase, the first ML algorithm generates class predictions based on input image data.

[0136] In some embodiments, the first machine learning algorithm may be a supervised learning algorithm, and a training phase may be performed to train the supervised learning algorithm as a classification algorithm, in which the supervised learning algorithm is trained with training data.

[0137] For example, the training phase may be designed according to any suitable scheme for training the first ML algorithm as a classifier using any suitable training data, such as, for example, a set of images annotated by medical personnel.

[0138] In some embodiments, several exemplary convolutional neural networks, such as Lenet5, vgg16, Mobilenet, Resnet, or I3D, may be tested to compare their performance in predicting the class of a given input ultrasound image, for example, based on a set of predefined classes.

[0139] An exemplary architecture of an apparatus, such as a computer system, according to the present disclosure is shown in FIG. 4, which illustrates an apparatus 1 configured to implement a method for ultrasound image processing according to an embodiment of the present disclosure.

[0140] The device 1 may include one or more computers, and includes a control engine 2, an ultrasound image processing engine 3, a data interface engine 4, a memory 5, and a prediction engine (not shown in FIG. 4).

[0141] In the architecture shown in FIG. 4, the ultrasound image processing engine 3, data interface engine 4, prediction engine, and memory 5 are all operatively coupled to each other through the control engine 2.

[0142] In some embodiments, the ultrasound image processing engine 3 is configured to implement various aspects of one or more of the embodiments of the proposed method for ultrasound image processing described herein, such as predicting the class of a resultant image using a first machine learning algorithm, and for predicting the detection of the presence in an input ultrasound image of a structure related to the procedure for observation based on a determination that the class is associated with a target organ, which has been determined based on a set of ultrasound images, i.e., should be used to generate an augmented image of the input ultrasound image.

[0143] In some embodiments, the data interface engine 4 is configured to receive an input set of ultrasound images, possibly as part of an input ultrasound video stream, and to output augmented reality data for generation of an augmented reality ultrasound image based on the input ultrasound images.

[0144] In some embodiments, the prediction engine may be configured to implement a first artificial intelligence algorithm that uses a neural network, such as, for example, a supervised learning algorithm. The prediction engine may further be configured to implement functionality or embodiments provided in this disclosure related to training or using the first artificial intelligence algorithm to obtain a classification prediction for input ultrasound image data.

[0145] The control engine 2 includes a processor, which may be any suitable microprocessor, microcontroller, field programmable gate array (FPGA), application specific integrated circuit (ASIC), digital signal processing chip, and / or state machine, or combination thereof. According to various embodiments, one or more of the computers may be configured as a multiprocessor computer having multiple processors to provide parallel computing. The control engine 2 may also include or be in communication with a computer storage medium, such as, but not limited to, a memory 5, capable of storing computer program instructions or software code that, when executed by the processor, cause the processor to perform the elements described herein. Furthermore, the memory 5 may be any type of data storage or computer storage medium coupled to the control engine 2 and operable with the data interface engine 4 and the ultrasound image processing engine 3 to facilitate management of data stored in association therewith, such as, for example, a cache memory, a data farm, a data warehouse, a data mart, a data center, a data cloud, or a combination thereof.

[0146] In embodiments of the present disclosure, device 1 is configured to perform one or more of the methods described herein, and in some embodiments may be included in a video processor or, in some embodiments, an FPGA component.

[0147] It will be appreciated that the device 1 shown and described with reference to FIG. 4 is provided for illustrative purposes only. Numerous other architectures, operating environments, and configurations are possible. Other embodiments of the node may include fewer or more components and may incorporate some or all of the functionality described with respect to the device components shown in FIG. 4. Thus, although the control engine 2, ultrasound image processing engine 3, data interface engine 4, prediction engine, and memory 5 are shown as part of the device 1, no constraints are imposed on the placement and control of these components. Notably, in other embodiments, any of these components may be part of a different entity or computing system.

[0148] Embodiments of the proposed method for encoding image data offer several advantages in the context of EUS for pancreatic imaging, including the following:

[0149] By proposing solutions to overcome the two main limitations of EUS, the number of early-stage pancreatic cancers identified can be advantageously increased, thus improving overall patient survival.

[0150] To address issues related to accurate image-based cancer diagnosis (cancer vs. no cancer, metastatic vs. normal nodes, etc.), the present disclosure advantageously provides embodiments of a system configured for artificial intelligence (AI)-based computer-assisted image recognition for EUS technicians. Depending on the embodiment, the proposed AI-based solution can advantageously be operated to facilitate lymph node detection, automatically estimate its malignancy, provide insight into predictors of pancreatic cancer, and also estimate the quality of the pancreatic parenchyma.

[0151] Depending on the embodiment, presence detection may be configured to provide lesion (e.g., solid, gallbladder, lymph node) detection and / or vessel type detection, which may be performed using an AI model (e.g., neural network) configured to generate detection prediction data based on input image data.

[0152] Depending on the embodiment, classification performed using an AI model (e.g., a neural network) configured to generate classification prediction data based on input image data may advantageously result in improved EUS operations, such as assisted pancreatic screening methods for legal protection in quality reports, and / or improve trainee EUS skills and increase self-confidence in the EUS profession.

[0153] Furthermore, the use of AI models configured as classifiers does not have any significant impact on system performance, particularly in response time or computational complexity, given state-of-the-art computer systems implementing AI techniques.

[0154] While the present invention has been described with reference to preferred embodiments, it will be readily apparent to those skilled in the art that various changes and / or modifications can be made to the present invention without departing from the spirit or scope of the invention as defined by the appended claims.

[0155] Although the present invention has been disclosed in the context of certain preferred embodiments, it should be understood that certain advantages, features, and aspects of the systems, devices, and methods may be realized in various other embodiments. Furthermore, it is contemplated that the various aspects and features described herein may be practiced separately, in combination, or substituted for one another, and that various combinations and subcombinations of features and aspects may be made and still fall within the scope of the invention. Furthermore, the systems and devices described above may not include all of the modules and functions described in the preferred embodiments.

[0156] The information and signals described herein may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0157] In some embodiments, some acts, events, or functions of any of the methods described herein may be performed in a different order, added, combined, or entirely excluded (e.g., not all described acts or events may be required to implement the method). Moreover, in some embodiments, acts or events may be performed in parallel rather than sequentially. [Explanation of symbols]

[0158] 1 device 2 Control Engine 3 Ultrasound image processing engine 4 Data Interface Engine 5. Memory 10 Systems 11 Ultrasound Image Acquisition Subsystem, Subsystem 12 Pre-processing Subsystem 12_2 Pre-treatment unit 12_7 AR data post-processing unit 12a Ultrasound imaging system, system 12a3_1 First Artificial Intelligence (AI) Model, First AI Model Unit, AI Model Unit 12a3_2 Second Artificial Intelligence Model, Second AI Model Unit, AI Model Unit 12a4_1 Aftertreatment unit 12a4_2 Aftertreatment unit 12b Ultrasound imaging system, system 12b3 AI Model Unit 12b4 Aftertreatment Unit 13 Augmented Reality Subsystem 14 Display Subsystem

Claims

1. 1. A method for processing a set of ultrasound images generated for viewing a target organ, comprising the steps of: determining a prediction of a class of the resulting image using a first machine learning algorithm, the class belonging to a set of classes including at least one class associated with the target organ; determining whether the class is associated with the target organ; and generating an output signal based on a determination that the class is associated with the target organ using a prediction of the detection of the presence in the resultant image of a structure related to a procedure for observing the target organ determined based on the set of ultrasound images.

2. The method of claim 1 , wherein the set of ultrasound images is generated by an ultrasound endoscope.

3. 3. The method of claim 1 or 2, wherein the detection of the presence in the result image of the structure related to the procedure for observing the target organ includes detecting the presence of tissue and / or lesions of the target organ in the result image.

4. 4. The method of claim 1, wherein the prediction of detection is determined by the first machine learning algorithm or by a second machine learning algorithm.

5. 5. The method of claim 1, wherein the prediction of detection is determined by a second machine learning algorithm, and the first and second machine learning algorithms run in parallel to generate the prediction of the class and the prediction of the detection based on the set of ultrasound images.

6. 6. The method of claim 1, wherein the first machine learning algorithm is configured to receive preprocessed images of the set of ultrasound images as input data.

7. 7. The method of claim 1, wherein the first machine learning algorithm is configured to implement an artificial intelligence algorithm using a neural network.

8. 8. The method of claim 1, wherein the prediction of detection is determined by a second machine learning algorithm, the second machine learning algorithm being configured to receive preprocessed images of the set of ultrasound images as input data.

9. The method of claim 1 , wherein the output signal comprises an augmented result image.

10. 10. The method of claim 1, wherein the first machine learning algorithm is a supervised learning algorithm, the method further comprising performing a training phase for training the supervised learning algorithm as a classification algorithm, in which the supervised learning algorithm is trained with training data.

11. The method of claim 9, wherein the augmented result image is generated by overlaying information in the result image related to the presence of tissue in the target organ and / or the presence of a lesion in the target organ based on a prediction of the detection of the presence of tissue and / or a lesion in the target organ in the result image.

12. The method of claim 9 or 11, wherein the set of ultrasound images is extracted from an ultrasound video stream generated by an ultrasound endoscope for observing the target organ, and the method further comprises the step of displaying an enhanced ultrasound video stream including the enhanced resultant images on a display.

13. 13. The method of claim 1, wherein the prediction of the class comprises a likelihood value associated with the class, and wherein determining whether the class is associated with the target organ is based on a comparison of the likelihood value with a predefined threshold.

14. 14. An apparatus comprising: a processor; and a memory operatively coupled to the processor, the apparatus configured to perform the method of any one of claims 1 to 13.

15. 14. A computer program product comprising computer program code tangibly embodied on a computer readable medium, said computer program code comprising instructions that, when provided to a computer system and executed, cause the computer to perform the method of any one of claims 1 to 13.

Images