Method and system for automatically detecting anatomical structures in medical images

Abstract

The present invention relates to a computer-implemented method for automatically detecting anatomical structures 3 in a medical image 1 of a subject, the computer-implemented method comprising applying an object detection function 4 to the medical image, the object detection function performing the following steps: (A) applying a first neural network 40 trained to detect a first plurality of classes of large-sized anatomical structures 3a to the medical image to generate, as output, coordinates of at least one first bounding box 51 and a confidence score that the first bounding box contains the large-sized anatomical structure; (B) cropping the medical image to the first bounding box to generate a cropped image 42 containing the image content within the first bounding box; and (C) applying a second neural network 44 trained to detect at least one second class of small-sized anatomical structures 3b to the cropped medical image to generate, as output, coordinates of at least one second bounding box and a confidence score that the second bounding box contains the small-sized anatomical structure.

Description

[Technical Field]

[0001] The present invention relates to a computer implementation method for automatically detecting anatomical structures in medical images of a subject, a method for training an object detection function effective for detecting multiple classes of anatomical structures in medical images, an associated computer program, and a system. [Background technology]

[0002] Medical imaging modalities such as X-ray, computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and ultrasound imaging are widely available and repeatedly used for diagnostic and other medical purposes. However, other factors such as various artifacts, noise, and image inaccuracies can arise. of Therefore, interpreting medical images remains challenging. Specifically, the interpretation of ultrasound images is known to be highly variable, both within and between users, even among experts such as radiologists and certified ultrasound technicians. This situation is exacerbated by the fact that ultrasound diagnoses are often performed by junior physicians or emergency room doctors in medical emergency situations. Thus, computer assistance is needed in detecting and identifying anatomical structures from medical images and / or determining the likelihood of a given medical condition.

[0003] One example where computer-assisted ultrasound is desirable is prenatal ultrasound, which is recommended worldwide for all pregnant women. The primary purpose of early pregnancy ultrasound scanning is to assess the viability of the pregnancy, date it, count the number of fetuses, and rule out abnormalities in the first trimester, such as ectopic pregnancy and miscarriage. Pregnancy loss is common in this early stage, and ectopic pregnancy, a significant clinical abnormality that often goes undetected, remains a major cause of maternal death worldwide. Ultrasound imaging plays a central role in identifying the aforementioned clinical findings early in the first trimester. However, even among experts, there is considerable variability between and within observers, which significantly limits the diagnostic value of ultrasound images.

[0004] Deep learning techniques have made remarkable progress in pattern recognition, object detection, image classification, and semantic segmentation in recent years. The first attempts involved applying artificial neural networks to pinpoint the location of anatomical structures in medical images.

[0005] The paper "ConvNet-Based Localization of Anatomical Structures in 3D Medical Images" by Bob D. de Vos, Jelmer M. Wolterink, Pim A. de Jong, Tim Leiner, Max A. Viergever, and Ivana Isgum, published in IEEE Transactions on Medical Imaging, PP. DUI.: 10.1109 / TMI. 2017. 2673121 on April 19, 2017, proposes a method for automatic localization in 3D medical images by detecting the presence of one or more anatomical structures in 2D image slices using a convolutional neural network (ConvNet). A single ConvNet is trained to detect the presence of anatomical structures of interest in axial coronal and sagittal slices extracted from a 3D image. Spatial pyramid pooling is applied so that the ConvNet can analyze slices of different sizes. After detection, a 3D bounding box is generated by combining the outputs of the ConvNet in all slices. The output feature map of the spatial pyramid pool layer is connected to a series of two fully connected layers, which in turn connect to 2N end-node output layers, where N indicates the number of anatomical structures in question. Spatial pyramid pooling enables the analysis of images with variable input sizes.

[0006] A method for object detection disclosed in WO 2017 / 1242221 A1 includes the steps of: grouping the detected object class into a plurality of object clusters constituting a hierarchical tree structure; obtaining an image and obtaining at least one bounding box for the obtained image; evaluating the objects in each bounding box from the root cluster to the leaf cluster of the hierarchical tree structure using a convolutional neural network trained for each cluster of the hierarchical tree structure, to determine the deepest leaf cluster of the object; and outputting the object class label in the determined deepest leaf cluster as the predicted object class label of the object.

[0007] The paper "A hierarchical model for automatic nuchal translucency detection from ultrasound images" by Y. Deng, Y. Wang, P. Chen, and J. Yu, Computers in Biology and Medicine, No. 42 (2012), pp. 706-713, proposes an algorithm for automatic detection of apical translucency (NT) regions. When ultrasound images are given, the entire fetal body is recognized and positioned first. Then, based on the body's characteristics, the NT region and the fetal head can be inferred from the image. An established graphical model appropriately represents such causal relationships between the NT region, head, and body objects.

[0008] The paper "Hierarchical part detection with deep neural networks" by Cervantes Esteve et al., presented at the 2016 IEEE International Conference on Image Processing (ICIP), IEEE, September 25, 2016, pp. 1933-1937, XP033016808, discloses an end-to-end hierarchical object and part detection framework. Thus, to detect an object, a bounding box is determined in the input image data using a single convolutional neural network. Furthermore, within the bounding box, one proposal is determined for each part of the object.

[0009] EP 2911111 A2 discloses an apparatus and method for lesion detection. The lesion detection method includes the steps of detecting lesion candidates from a medical image, detecting anatomical objects from the medical image, verifying each lesion candidate based on anatomical context information including information on the spatial relationship between the lesion candidate and the anatomical object, and removing one or more false-positive lesion candidates from the detected lesion candidates based on the verification results. [Overview of the project] [Problems that the invention aims to solve]

[0010] An object of the present invention is to provide a robust computer-aided method for detecting and identifying anatomical structures in medical images, preferably operating at near real-time speeds on general-purpose hardware. A further object of the present invention is to provide a computer-aided solution for identifying a specific anatomical structure with accuracy equivalent to or exceeding that of a human. As an example, it is desirable to obtain a robust computer-aided solution that improves the OBGYN workflow (diagnosis of IUP and its gestational age) for identifying the features of intrauterine pregnancy (IUP) and its abnormal fraternity (specifically pregnancy loss, ectopic pregnancy). [Means for solving the problem]

[0011] These objectives are met or exceeded by the computer implementation method according to claim 1 for automatically detecting anatomical structures in medical images of a subject, the method according to claim 11, the computer program according to claim 14, and the system according to claim 15 for training an object detection function effective for detecting multiple classes of anatomical structures. Advantageous embodiments are detailed in the dependent claims. Any features, advantages or alternative embodiments described herein in relation to the claimed method are also applicable to other categories of claims and aspects of the invention, and more specifically to the training method, the claimed system and computer program, and vice versa. More specifically, the claimed method may provide or improve upon object detection functions and neural networks. Furthermore, input and output data for object detection functions may include advantageous features and embodiments of input and output training data, and vice versa.

[0012] According to a first aspect of this invention, a computer-aided method is provided for automatically detecting anatomical structures in medical images of a subject, and this computer-aided method is a) A step of receiving at least one medical image of the subject's FOV (field of view), b) A computer-aided method comprising the steps of applying an object detection function to a medical image, thereby generating the coordinates of multiple bounding boxes and a confidence score for each bounding box as output, wherein the object detection function is trained to detect multiple classes of anatomical structures, and the confidence score indicates the probability that the bounding box contains an anatomical structure belonging to one of the multiple classes, wherein the object detection function - A step of applying a first neural network trained to detect large-sized anatomical structures of a first class to a medical image, thereby generating as output the coordinates of at least one first bounding box and a confidence score that this first bounding box contains a large-sized anatomical structure. - A step of cropping a medical image to a first bounding box to generate a cropped medical image containing the image content within the first bounding box, - A step of generating, as output, the coordinates of at least one second bounding box and a confidence score that this second bounding box contains a small-sized anatomical structure, by applying a second neural network trained to detect at least one second class of small-sized anatomical structures to a cropped medical image. Execute This is a computer implementation method characterized by the following features.

[0013] Therefore, the present invention provides a hierarchical anatomical identification mechanism for automatically detecting anatomical structures of various sizes or detail scales in medical images. The present invention assumes that certain anatomical structures of smaller size or scale detail (small-size anatomical structures) are expected to be found within other anatomical structures of larger size or scale detail (large-size anatomical structures). The input medical image is favorably cropped to a bounding box containing the large-size anatomical structure, and the cropped image is used to search for the smaller-size anatomical structure. Thus, the neural network used to detect anatomical structures at each hierarchical level (e.g., large-size and small-size levels) requires a very simple architecture, can be trained faster, is more robust, and thus has higher average accuracy. In other words, an independent, separate neural network is implemented at each hierarchical level and is therefore specifically trained for a particular detection task according to that hierarchical level. As an example, in an early pregnancy ultrasound (US) image, the yolk sac is expected to be found within the gestational sac. However, it has been found that the yolk sac (YS) is a very fine tissue and cannot be well trained when combined with relatively large anatomical structures. To achieve better detection, a dedicated second neural network is trained on images, for example, the gestational sac (GS) trimmed from the original input medical image. This reduces the search area, thereby improving training and detection.

[0014] In one embodiment, the computer implementation method of the present invention is performed on any general-purpose hardware such as a regular PC, laptop computer, tablet computer, cloud computer, or server, and more specifically, on an ultrasound system used to perform ultrasound scanning. This method is also performed on a remote computer, i.e., images acquired by ultrasound scanning are uploaded to a remote computer or server, for example, via the internet or a cellular connection, where they are processed according to the present invention. Results of the present invention, such as the coordinates of at least one first bounding box and at least one second bounding box containing a particular anatomical structure, the confidence score of these bounding boxes, and generally the class of the anatomical structure, can be sent back to the ultrasound scanner or other hardware device via the internet or a cellular connection and used for image evaluation.

[0015] This invention utilizes trained artificial neural networks, namely a first neural network and a second neural network (NN) used in succession. That is, the first and second neural networks may have different inputs and / or different outputs. In this way, a hierarchical and anatomical recognition mechanism is implemented for a given anatomical structure, consistently demonstrating excellent results in the detection of predetermined anatomical structures. This allows for a more systematic diagnostic approach. In the case of the first trimester of pregnancy, this invention fundamentally improves the clinical outcomes of abnormal early first trimesters of pregnancy, particularly by improving the detection rate of ectopic pregnancies. However, this invention is not limited to early pregnancy and is applicable to a variety of medical conditions and clinical applications.

[0016] The medical images input to the object detection function are preferably 2D images. This is because the preferred first and second neural networks are best suited for 2D image processing. However, extension to 3D is possible. The input medical images are generated by any medical imaging modality such as X-ray, CT, MRI, PET, or ultrasound such as B-mode ultrasound, color Doppler, or shear wave elastography.

[0017] In one embodiment, this method is applied to a series of medical images, such as a time series of medical images acquired from a moving target tissue, such as the heart. This method is also applied to a series of medical images covering various fields of view, such as those obtained during ultrasound scanning by sweeping the probe from edge to edge of the region of interest. In one embodiment, this method renders the results in real time or near real time, for example, at a frame rate of 10-60 FPS (frames per second), preferably 20-40 FPS, in order to process a large number of medical images.

[0018] The field of view (FOV) of the input medical image covers any region of interest of the human or animal body, such as the head or brain, limbs, parts of limbs, or any organ or group of organs inside, such as the heart, lungs, breasts, liver, kidneys, reproductive organs, or intestines, within, for example, the chest, torso, or abdomen. An "anatomical structure" is any identifiable anatomical feature within such an FOV, such as any of the aforementioned organs or parts thereof, such as the uterus, gestational sac, embryo, fluid in the pouch of Douglas, ovaries, yolk sac, specific bones, blood vessels, or heart valves; or abnormal tissues such as tumors, cysts, lesions, or aneurysms; or transplanted tissues such as screws, knee or shoulder implants, or transplanted heart valves. In one embodiment, the object detection function is trained to detect several different predefined anatomical structures, each type of anatomical structure corresponding to a "class" (e.g., "uterus" is one class, and "ovaries" is another). In one embodiment, each predetermined type / class of anatomical structure is bounded relative to other organs in the medical image so that a radiologist can draw a bounding box that completely includes the anatomical structure. In one embodiment, the bounding box generated by the object detection function is rectangular and preferably aligned with axes, i.e., the four sides of the rectangle are aligned with the four edges of the medical image and / or the sides of other bounding boxes.

[0019] In one embodiment, each input medical image is a square 2D image, i.e., having an equal number of pixels in the width and height directions. For this invention, it is advantageous that the object detection function is trained for a specific image size (e.g., 416 × 416 pixels). Therefore, the method of the present invention has a further optional step in which the medical image received from the image diagnostic method is rescaled to an input size defined with respect to the object detection function, for example by known interpolation techniques.

[0020] The object detection function includes a first neural network trained to detect large-sized anatomical structures and a second neural network trained to detect small-sized anatomical structures. "Large size" and "small size" mean that the small-sized anatomical structure is generally (i.e., in most subjects) smaller than the average of the large-sized anatomical structure or has more detailed features. In one embodiment, at least one of the small-sized anatomical structures is generally contained within or is a part of one of the large-sized anatomical structures. In terms of actual size, the small-sized anatomical structure generally has an average diameter of less than 40 mm, preferably less than 30 mm, most preferably less than 20 mm, and optionally, the minimum size is 0.2 to 5 mm. The large-sized anatomical structure preferably has an average diameter generally greater than 10 mm, preferably greater than 20 mm, more preferably greater than 30 mm, and optionally has a maximum size of 50 to 400 mm. Examples of corresponding sets of large-sized and small-sized anatomical structures are shown in the table below.

[0021]

Table 1

[0022] The object detection function is based on the hierarchical relationship between large-sized anatomical structures and small-sized anatomical structures, and at least one of the large-sized anatomical structures of the first plurality of classes is expected to include one or some of the small-sized class of anatomical structures in at least some medical conditions. In the first trimester screening, for example, the large-sized anatomical structures of the class include the uterus (U), the gestational sac (GS), and optionally the embryo (E). The small-sized anatomical structures of the class include the yolk sac (Y) and optionally the embryo (E). Thereby, the method of the present invention enables the automatic detection of IUP (intrauterine pregnancy) based on the relationship of each bounding box that locates the uterus, GS, yolk sac, and embryo using a deep learning-based method automated by generating bounding boxes. Since both the uterus and GS are of the large-sized anatomical structures of the first plurality of classes, the first level of automation (i.e., the first neural network) locates the presence of GS inside or outside the uterus. In the next step, the medical image is trimmed to a first bounding box which is a bounding box containing the gestational sac in one example, thereby generating a trimmed image containing the image content, i.e., a small image mainly containing only GS, inside the first bounding box. The second level of automation (using the second neural network) identifies the presence / absence of the embryo box and the yolk sac box inside the trimmed image, i.e., inside the gestational sac. This hierarchical relationship of the bounding boxes enables the automatic identification of IUP and non-IUP such as ectopic pregnancy.

[0023] The output of the method of the present invention is a bounding box and a confidence score for each bounding box containing a particular class of anatomical structure. This output is visualized by outputting the medical image and / or the trimmed image, and displaying the bounding box having the highest confidence score, for example, in a contrasting color scheme or overlaid.

[0024] An advantage of the present invention is that it may enable computer-aided, robust identification of both large-sized anatomical structures and at least one, usually several, smaller-sized anatomical structures that may be contained within one of the larger-sized anatomical structures. By using this two-step method, the detection and identification of smaller-sized anatomical structures becomes significantly more precise and robust. In the case of a field of view (FOV) covering a woman's uterus in early pregnancy, the present invention helps to more systematically diagnose normal / abnormal features in early pregnancy, particularly ectopic pregnancy, molar pregnancy, or adnexal features.

[0025] In one embodiment, the method includes a further step c) determining the probability of a subject having a predetermined medical condition using an inference scheme based on the presence or absence of one or more classes of anatomical structures and / or the relative spatial location of a detected boundary box containing the anatomical structures.

[0026] A given medical condition is a clinical finding that can be inferred from the presence or absence and relative spatial location of a detected boundary box containing an anatomical structure. The clinical finding itself is usually not a diagnosis, but simply a probability of a particular medical condition, such as the probability of a tumor or other local abnormality (which may be a small anatomical structure) in a given organ (which may be a large anatomical structure), intrauterine pregnancy, ectopic pregnancy, or pregnancy loss (for example, when no embryo or yolk sac is found inside the gestational sac).

[0027] The expressed probability can be 1 or 0, but an intermediate value may be adopted, and the decision may depend on the confidence score of the bounding box used. The inference scheme includes multiple IF ELSE commands, for example, implemented by an algorithm in any computer language. In the case of the first trimester of pregnancy, the inference scheme represents the uterine bounding box as Bu and the GS bounding box as BGS. If BGS is a subset of Bu, it is presumed to be a normal IUP; otherwise, it is called an abnormal pregnancy. In a second level of automation using a second neural network, the yolk sac (YS) and embryo are detected and located. The yolk sac bounding box is called BYS and the embryo bounding box is called BE. If BYS and / or BE are a subset of BGS, it is called a normal pregnancy. If YS and embryo are not detected within GS, the probability of an abnormal pregnancy is increased. The "increased" probability of a medical condition means that it exceeds a predetermined threshold, for example, a value of 50-95%, preferably 60-90%.

[0028] Therefore, in some embodiments, the probability of a predetermined medical condition (e.g., IUP) is increased if a first detection bounding box containing a first class of anatomical structures (e.g., GS) encompasses or contains a second detected bounding box containing a second class of anatomical structures (e.g., GS). The first anatomical structure completely encompasses, i.e., fully contains, the second bounding box, or, if applicable, the algorithm allows a predetermined amount of overlap. For example, in order to increase the probability, a predetermined proportion of the second bounding box (e.g., at least 80%) must be inside the first bounding box.

[0029] In other embodiments, the relative spatial position of each bounding box to be detected may not only require that one bounding box be a subset of other bounding boxes, but may also require that the bounding boxes of two or more classes of anatomical structures have a size ratio within a predetermined range, a predetermined amount of overlap, and / or a predetermined spatial relationship.

[0030] According to one embodiment, this method is repeated for multiple 2D medical images with separate FOVs acquired during the same consultation period of a subject, and the confidence scores of the detected bounding boxes are used to calculate the optimal medical image or FOV for further evaluation. In this case, further evaluation is either performed by an expert user to further observe each medical image with the highest confidence score bounding box, or subjected to further automated image analysis techniques such as segmentation, modeling, feature detection, distance measurement, or feature tracking. Further evaluation also means that, if the medical images used so far are B-mode US images, the identified FOV may be used by other imaging techniques, such as Doppler ultrasound, to acquire further images from this FOV.

[0031] In another embodiment, the orientation of a further 2D plane used to acquire further medical images is also determined from the detected bounding box. Thus, the detected bounding box is used, along with the corresponding input image, to calculate the best plane, which means, for example, the one with the highest confidence score. For example, a weighted average of the bounding box and confidence score is used to determine a confidence interval, which is then used to calculate the best plane. For example, a medical image capturing the largest cross-section of a particular anatomical structure is used for further evaluation. Then, in the example of first trimester screening, these planes / images, along with the corresponding bounding boxes, are used for the automatic calculation of key parameters such as, for example, the diameter of the yolk sac, the mean sac diameter of the gestational sac, the crown-rump length of the embryo, ectopic / ectopic mass, and the volume of fluid in the cul-de-sac of Douglas. Thus, the present invention can also be used to automatically identify either a standard plane / FOV or a superior plane / FOV, from which further measurements are performed on the anatomical structure.

[0032] The object detection function according to the present invention is provided in the form of a software program, but can also be implemented in hardware. In one embodiment, each detection step of the object detection function is performed by a single neural network. In the first step, one first NN is applied to the entire medical image, and in the second step, one second NN is applied to the cropped image. Each NN divides the input image into regions and predicts bounding boxes and probabilities for each region. These bounding boxes are weighted by the predicted probabilities. In one embodiment, only bounding boxes with a certain confidence score of 20-40%, for example, 25% or more, are displayed and / or taken into consideration.

[0033] In one embodiment, a first artificial neural network (NN) and a second NN, which form part of the object detection function, have similar or identical architectures but are trained to detect different classes of anatomical structures, and possibly different numbers of classes. The second NN is more specialized to detect specific, smaller-sized anatomical structures that are more difficult to detect in the complete medical images input to the first NN. The first NN and the second NN are also different types. Therefore, wherever the term “neural network” is used below, it always means the first NN and / or the second NN, and, though not mandatory, preferably both.

[0034] An artificial neural network (NN) is based on a collection of connected artificial neurons, also called nodes, where each connection, also called an edge, can transmit a signal from one node to another. Each artificial neuron processes the received signal and then transmits it to further connected artificial neurons. In a valid embodiment, the artificial neurons of the first NN and the artificial neurons of the second NN are configured in layers. Input signals, which are pixel values ​​of a medical image, are transmitted from the first layer, also called the input layer, to the final layer, which is the output layer. In one embodiment, the first NN and / or the second NN are feedforward networks. The first NN and the second NN preferably each have several layers, including hidden layers, and are therefore deep neural networks. In one embodiment, the first NN and the second NN are trained based on machine learning techniques, specifically deep learning such as backpropagation. Furthermore, the first NN and the second NN are provided in the form of software functions, which are not necessarily constructed in exactly the same way as a trained neural network. For example, if a particular connection or edge has a weight of 0 after training, such a connection will be omitted when providing an object detection function.

[0035] In one embodiment, the first neural network and / or the second neural network do not have fully connected layers, where each node is connected to each node in a subsequent layer. In one embodiment, the first NN and / or the second NN comprises at least one convolutional layer. In one embodiment, the first and / or the second neural network are preferably both full convolutional neural networks (CNNs). A full convolutional NN is defined as a convolutional NN that does not contain fully connected layers. The convolutional layer applies a relatively small filter kernel across all layers such that neurons within the layer are connected only to small regions of the layer being advanced. This architecture ensures that the trained filter kernel produces the strongest response to spatially local input patterns. In one embodiment of the present invention, the parameters of the convolutional layer include a set of trainable filter kernels, which have a small perceptual field but extend to the full depth of the layer volume. Through a forward pass through the convolutional layer, each filter kernel is convolved across the width and height of the input layer, and the dot product between the filter kernel's entry and the input layer is calculated to generate an output map for its own filter kernel. Stacking the output maps of all filter kernels along the depth dimension forms the total output volume of the convolutional layer, also referred to herein as the feature map.

[0036] A convolutional layer is generally defined by the size and stride of its filter kernel. Several filter kernels within a single convolutional layer each generate different output maps, and these output maps are stacked together along the depth dimension for all feature kernels, forming an output volume or feature map. Filter kernels are typically spread across the entire depth of the input volume. Therefore, if the input layer size for a convolutional layer is 416×416×3 and the filter kernel size is 3×3, it essentially means the convolutional filter kernel size is 3×3×3. This results in one feature map. The stride of the filter kernel is the number of pixels, and the filter kernel shifts by this number when convolving around the input layer / volume. Therefore, if the filter kernel stride is 2, the size of the output layer will be half that of the input layer.

[0037] In one embodiment, the first NN and / or the second NN comprises a block of 3 to 14 layers, preferably 4 to 10 layers, each block comprising a convolutional layer using multiple filters, each having a filter kernel of size 3x3 and stride 1, followed by a maxpool layer of size 2x2 and stride 2. Thus, each such block halves the size of the layers. In one embodiment, the convolutional layers of the first NN and / or the second NN process the input medical image in a 2 3 ~2 7 The coefficient is preferably 2 4 ~2 6 For example, the coefficient of 2 5 Downsampling is performed with a coefficient of =32. In this example, by using an input image of size 416×416, the output feature map will have a size of 13×13.

[0038] By using a combination of convolutional and maxpool layers, the image is reduced in size as it propagates through the first and / or second NN, thereby ultimately yielding a 3D tensor encoding the coordinates of several bounding boxes, a confidence score for each bounding box, and class predictions, such as the probability that the detected object belongs to one of a predetermined class of anatomical structures.

[0039] In one embodiment, the first and / or second neural networks are a YOLOv3 network, more specifically a modification of a miniature YOLOv3 network. YOLOv3 is disclosed in J. Redmon and A. Farhadi's "YOLOv3: An Incremental Improvement," arXiv preprint arXiv: 1804.02767, 2018, https: / / arxiv.org / abs / 1804.02767. When trained against a miniature version of YOLOv3, detection works effortlessly without the need for any additional hardware assistance.

[0040] Output of the First NN / Second NN: In one embodiment, the first NN and / or the second NN predict, for each possible bounding box, a confidence score ("object-likeness") indicating the probability that the bounding box contains an anatomical structure belonging to one of several classes, and the probability that the object (anatomical structure) within the bounding box belongs to each of the trained classes, i.e., the class probability. If the network is trained on 10 different classes of objects / anatomical structures, it predicts 10 probability values ​​for each bounding box. Only bounding boxes whose confidence score and class probability exceed a certain predetermined threshold are taken into consideration.

[0041] The final output of the first NN and / or second NN of the present invention is generated in one embodiment by applying a 1×1 detection kernel to a (final) downsampled feature map, also referred to as the output grid, and in one embodiment to two feature maps of two different sizes at two different locations in the network. The shape of the detection kernel is 1×1×(B*(5+C)), where B is the number of bounding boxes on the feature map that the grid cells can predict, "5" is for four bounding box attributes (offsets in the x, y directions and width / height offsets from the anchor box, as described below), and one object confidence score, also referred to as "object likelihood," and C is the number of classes. Thus, if the final feature map is N×N (e.g., 9×9, 13×13, or 26×26), the 3D output tensor has a size of N×N×[B*(4+1+C)] with respect to the anchor box of B, four bounding box attributes, one object likelihood prediction, and a class prediction of C. If B=3, each grid cell in the final layer predicts up to three bounding boxes (corresponding to three anchor boxes). For each anchor box, the tensor contains a confidence score, i.e., the probability that the box contains the object, four numbers representing the coordinates of the bounding box relative to the anchor box, and a probability vector containing the probability that the object belongs to one of the respective predefined classes. Each class score is predicted using logistic regression from different class probabilities, and a threshold is used to predict one or more annotations for each detected anatomical structure. Class probabilities higher than this threshold are assigned to bounding boxes.

[0042] Anchor Box / Bounding Box Plurias: In one embodiment, as in YOLOv3, the NN predicts the position of the bounding box by the xy offset relative to a specific cell in the output grid (e.g., a 13x13 grid). Once the image is divided into the final grid, for each object (anatomical structure), the grid cell containing the center of the object is identified, and this grid cell then "involves" in the prediction of the object. Thus, the center of each bounding box is described by the offset from the involved cell. Furthermore, instead of directly predicting the width and height of the bounding box, the NN in this embodiment predicts the width and height offset relative to the previous box, also referred to herein as the anchor box (as in YOLOv3). Thus, during training, the network is trained to predict the offset from a given set of anchor boxes having a specific height-to-width ratio determined from the training data using clustering. The coordinates of the annotation boxes in the training data are clustered to the number of anchor boxes as needed; for example, a small YOLOv3 requires six anchor boxes. Typically, a k-means clustering algorithm is used to generate the set of anchor boxes. The ratio of the intersection to the union (IOU) between the ground truth and the anchor boxes is typically interpreted as the distance metric for the k-means clustering algorithm. In one embodiment of the first and / or second NN of the present invention, each grid cell in the output tensor predicts three bounding boxes by their height and width offsets from three anchor boxes. In other words, if the size of the output grid is 13x13, then up to 13x13x3 bounding boxes can be detected.

[0043] Inter-scale detection: According to one embodiment of the present invention, a first NN and / or a second NN include detecting anatomical structures at two to three, preferably two different scales, each scale being given by predetermined downsampling of a medical image. This concept is named “inter-scale prediction” by the authors of YOLOv3. Thereafter, an early layer in the NN is combined with a later layer (which is upsampled first) using concatenation. The reason for this is that smaller objects are more easily detected at a higher resolution in the early layer than in the later layer, which is lower resolution and heavily downsampled, while the later layer contains semantically strong features. With this method, by combining the higher resolution early layer with the upsampled feature map, it becomes possible to obtain more meaningful semantic information from the upsampled feature map and finer-grained information from the early feature map. Thereafter, a second output tensor having twice the size of the prediction at scale 1 is predicted. In YOLOv3, this is done on three scales, but in the network of the present invention, prediction is preferably made on only two different scales, one on the most downsampled layer and the other on the final layer. 1 ~2 2 It has twice the size. Generally, the neural network of the present invention predicts (N×N + 2N×2N)×3 bounding boxes for two different scales. The bounding boxes are filtered using a threshold and class probability for confidence scores, and according to one embodiment, another filter called non-maximal suppression is applied, which is a function of the ratio (IOU) of the intersection and union of two bounding boxes. The main step of the maximum suppression filter is, Select a bounding box (that satisfies the threshold for class probability and confidence score), Calculate the overlap with all other boxes that meet the threshold, and remove any boxes with an overlap greater than or equal to a given IOU threshold. Return to step a) and repeat until there are no boxes with a lower confidence score than the currently selected box.

[0044] Further information on non-maximal suppression can be found at https: / / arxiv.org / pdf / 1704.04503.pdf.

[0045] This ensures that the best bounding box, specifically the best bounding box between the two scales, remains as the output.

[0046] A major advantage of using this network architecture is that, as can be realized with regard to clinical applications, both the training and robustness of the network are greatly improved, especially when the network is trained for only a limited number of classes.

[0047] The size of the input medical image is preferably an odd multiple of 2Z, where Z is an integer between 4 and 10, so that after several (e.g., Z) downsampling steps, the final grid will be an odd size such as 7x7, 9x9, 11x11, or 13x13. This is advantageous because there is one central grid cell, and in many cases, larger objects / anatomical structures are found in the center of the image, and consequently, there is one grid cell involved in detecting the largest object.

[0048] According to a preferred embodiment, the object detection function is trained to detect 2 to 12 classes, preferably 3 to 6 classes, of anatomical structures. For example, the first neural network is trained to detect 2 to 10 classes, preferably 3 to 6 classes. For example, in the case of first seminal screening, the classes are GS, uterus, and embryo. The second neural network is trained to detect even fewer classes, for example 1 to 4 classes, or just one class, for example, the yolk sac. By reducing the number of classes in this way, the first NN and the second NN can be very small, each with only 9 to 16 convolutional layers, preferably 13 convolutional layers, and thus can be very fast and robust.

[0049] According to one embodiment, step a) of the method includes receiving a video stream of medical images acquired during an ultrasound scan of a subject. Thus, the method of the present invention is applicable to time series of medical images at a frame rate of 20 to 100 frames per second. Furthermore, when ultrasound images are encoded into a video stream, they become color images with three channels rather than grayscale images. Generally, since ultrasound images are grayscale images, all three channels have the same values. Therefore, the input to the object detection function becomes a medical image in a three-channel format such as RGB. This has the advantage that the same NN architecture used for photographic images can be adapted for processing medical images.

[0050] According to its effective application, the medical image is acquired during a prenatal first semi-annual ultrasound scan, and multiple classes of anatomical structures include the uterus, gestational sac (GS), embryo, and / or yolk sac (YS). Most preferably, the first NN is trained to detect the first multiple classes, larger anatomical structures including the uterus, gestational sac, and embryo. Then, the second NN is trained to detect the yolk sac and, optionally, smaller anatomical structures.

[0051] Referring to the inference scheme, in one embodiment relating to the use of “first-term screening,” the probability of a “normal pregnancy” in medical condition is increased if the detected uterine boundary box includes the detected gestational sac boundary box, and the detected gestational sac boundary box includes the detected embryo and / or yolk sac boundary box. This hierarchical inference scheme of boundary boxes automatically distinguishes between normal IUPs and non-IUPs such as ectopic pregnancies. This makes the diagnostic method more systematic and allows for the use of a simpler method for detecting abnormal pregnancies.

[0052] In another aspect, the present invention provides a method for training an object detection function comprising a first neural network, which is effective in detecting multiple classes of anatomical structures in medical images, the method comprising: (a) The step of receiving input training data, i.e., at least one medical image of the subject's FOV, (b) Steps of receiving output training data, i.e., a tensor having coordinates of at least one first bounding box in a medical image containing a large-size anatomical structure belonging to one of a first class of large-size anatomical structures, and a number indicating the large-size class of anatomical structures, (c) A step of training the first neural network function using input training data and output training data. It has.

[0053] The training method is used to train the object detection function described herein, preferably the first neural network to be trained is constructed as described herein. The input training data is medical images of FOV as described herein, for example, B-mode ultrasound images of the human body, specifically the genitals in the first trimester of pregnancy. To generate output training data, the medical images constituting the input training data are manually annotated with axis-aligned bounding boxes, with each bounding box completely covering one anatomical structure. The data generated therefrom is a tensor having the coordinates of at least one first bounding box containing a large-sized anatomical structure, and a number indicating the class of the anatomical structure, for example, different classes used are uterus, GS, and embryo. In one embodiment, the object detection function is trained on a dataset derived from 5 to 15 subjects, with approximately 100 to 1000 images per subject. Bounding boxes are drawn for all possible anatomical structures present in these images. The training steps are carried out similarly to those of the darknet framework developed at https: / / pjreddie.com / darknet / yolo / . However, the placement parameters are preferably adjusted to achieve better performance in terms of training speed and training loss. In particular, the learning rate and batch size are preferably adjusted. In one embodiment of the training method, the following parameters are used: Number of batches during training = 64 Number of subdivisions during training = 16 Maximum batch size = 500200 Number of filters in the final layer = (number of classes + 5) × 3 Anchor point = Depends on image resolution and bounding box size. Number of steps = 24000~27000

[0054] In the training phase of the first and / or second neural networks, a batch of images is typically read and passed as input training data. Each batch is divided into minibatches. Let N be the batch size and n be the number of minibatches. Thus, the network is supplied with (N / n) images at a time. The parameters depend on the effectiveness of the GPU. Using smaller subdivisions increases the minibatch size for calculating gradients. Therefore, gradients calculated based on larger minibatch sizes lead to better optimization. A single batch may also be considered as an epoch, according to deep learning conventions.

[0055] According to one embodiment, the training method also includes a method for training a second neural network. (d) The step of receiving a cropped image which includes input training data, i.e., image content of a first bounding box containing a large-sized anatomical structure, (e) The step of receiving output training data, i.e., a tensor having the coordinates of at least one second bounding box inside a trimmed image containing small-sized anatomical structures belonging to at least one second class of small-sized anatomical structures, (f) A step of training the first neural network function using input training data and output training data. It further possesses.

[0056] Therefore, the same input training data may be used for training the first NN, but in this case, cropped images containing bounding box image content with larger anatomical structures are used as input training data. The output training data consists of the bounding box coordinates of smaller anatomical structures.

[0057] The training method described herein is used to provide object detection capabilities used to detect anatomical structures, i.e., to first train a first NN and / or a second NN. This training method is also used to recalibrate already trained networks. The training of the first and second NNs is performed by backpropagation. In this method, input training data is propagated through each NN using a predetermined filter kernel. The output is compared to the output training data using the error or the resulting capability, and the output is propagated back through the NN, thereby calculating the gradient to find the filter kernel, or other parameters such as a bias that gives the smallest error if applicable. This is done by adjusting the weights of the filter kernel and the subsequent negative gradient of the cost function.

[0058] The present invention also relates to a computer program containing instructions, and a computing unit performs the method of the present invention by executing the program. This applies to a method for automatically detecting anatomical structures in medical images and a training method, in particular to training a first NN and a second NN. The computer program is implemented using Darknet. Darknet is a framework developed for training neural networks, is open source, written in C / CUDA, and serves as the basis for YOLO. The repository and wiki are available at the link (https: / / pjreddie.com / darknet / ). The computer program is provided as a computer program product.

[0059] The computing unit that implements the method of the present invention is any processing unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The computing unit may be part of a computer, cloud, or server, or it may be a portable device such as a laptop computer, tablet computer, or mobile phone. More specifically, the computing unit may be part of an ultrasound imaging system. The ultrasound imaging system also includes a display device such as a computer screen.

[0060] The present invention also relates to a computer-readable medium containing instructions, and a computing unit executes the methods according to the present invention, more specifically, a method for automatically detecting anatomical structures in medical images or a training method, by executing the instructions. Such computer-readable media are, for example, hard disks, servers, clouds, or computers, and any digital storage medium, such as optical or magnetic digital storage media, CD-ROMs, SSD cards, SD cards, DVDs, USBs, or other memory sticks. Computer programs are stored on the computer-readable medium.

[0061] In one embodiment, the method also includes the step of displaying a medical image together with at least one first bounding box and at least one second bounding box displayed in contrasting colors, so that the user can inspect the accuracy of the prediction.

[0062] In another aspect, the present invention also relates to a system for automatically detecting anatomical structures in medical images of a subject, wherein the system a) A first interface configured to receive at least one medical image of the subject's FOV, b) A computing unit configured to apply an object detection function to a medical image, thereby generating the coordinates of multiple bounding boxes and a confidence score for each bounding box as output, wherein the object detection function is trained to detect multiple classes of anatomical structures, and the confidence score indicates the probability that the bounding box contains an anatomical structure belonging to one of the multiple classes, and this computing unit, - A step of applying a first neural network trained to detect large-sized anatomical structures of a first class to a medical image, thereby generating as output the coordinates of at least one first bounding box and a confidence score that this first bounding box contains a large-sized anatomical structure. - A step of generating a cropped image containing the image content within a first bounding box by cropping the medical image into a first bounding box, - A step of generating, as output, the coordinates of at least one second bounding box and a confidence score that this second bounding box contains a small-sized anatomical structure, by applying a second neural network trained to detect at least one second class of small-sized anatomical structures to a cropped medical image. It is configured to perform the following:

[0063] The system is preferably configured to perform the method of the present invention for automatically detecting anatomical structures in medical images. The computing unit is any processing unit as described above in relation to the computing unit that executes the program. The system is implemented in an ultrasound imaging system, more specifically in one of the processing units of the ultrasound imaging system, such as a GPU. However, the medical images are also transmitted from the imaging system to another local or remote computing unit, for example via the Internet, and the bounding box coordinates, as well as the probability of a predetermined medical condition, are also sent back to the imaging system and displayed or output to the user. In one embodiment, the system includes a second interface for outputting bounding box coordinates, more specifically for outputting a medical image with first and second bounding boxes drawn on it. Thus, the second interface is connected to a display device such as a computer screen or touchscreen.

[0064] Furthermore, the present invention also covers systems for training object detection functions, and more particularly, systems for training a first NN and / or a second NN by the training methods described herein.

[0065] In a further embodiment, the present invention relates to an ultrasonic imaging system comprising an ultrasonic transducer configured to transmit and receive ultrasonic signals, as described herein, and a computing unit configured to apply an object detection function to a medical image, wherein the ultrasonic imaging system comprises the system according to the present invention. The method of the present invention is computationally inexpensive and can therefore be performed on an existing ultrasonic imaging system.

[0066] Next, effective embodiments of the present invention will be described with reference to the attached drawings. Similar elements or features are indicated by the same reference numerals. [Brief explanation of the drawing]

[0067] [Figure 1] This is a medical image of an ultrasound scan with annotated bounding boxes of a fetus at 8 weeks and 4 days of gestation in the subject. [Figure 2] This is a flowchart illustrating one embodiment of the detection method according to the present invention. [Figure 3] This is a flowchart of another embodiment of the detection method according to the present invention. [Figure 4] This is a flowchart of an inference method based on one embodiment of the present invention. [Figure 5] This figure shows an example of anatomical structure positioning achievable using one embodiment of the present invention. (a) shows boxes around the uterus, gestational sac (GS), and embryo, and (b) shows a bounding box around the yolk sac (YS) in a cropped GS image. [Figure 6] This is a schematic diagram of the first and / or second neural network (NN). [Figure 7] This is a flowchart of the training method using one embodiment of the present invention. [Figure 8] This is a schematic diagram of a system according to one embodiment of the present invention. [Modes for carrying out the invention]

[0068] Figure 1 shows a possible training image for training object detection functionality, namely a 2D B-mode medical ultrasound image 1 acquired during a first trimester scan at 8 weeks and 4 days of gestation. Bounding boxes were drawn by the subject and annotated to generate output training data. The largest bounding box is drawn around the uterus (U), and another bounding box is drawn around the gestational sac (GS), with both the embryo (E) and yolk sac (YS) visible inside the gestational sac, in contrast to a situation where the embryo is not visible inside the GS, indicating a higher probability of a normal pregnancy.

[0069] Figure 2 shows one embodiment of a method for detecting anatomical structures in a series of 2D ultrasound images 1a, 1b, 1c, etc., in medical images 1. Each of these images covers a slightly different FOV 2, and organs, i.e., anatomical structures 3, are identifiable in image 1. These images are passed one by one, but the object detection function 44 will be described in more detail below. The object detection function 4 preferably comprises at least two neural networks 40, 44, as described herein. The output of the object detection function 4 is at least one bounding box 5 or rather its coordinates, as well as a confidence score that the bounding box 5 contains a particular anatomical structure. The confidence score is object likelihood, i.e., the probability that box 5 contains an object / anatomical structure, and / or the probability that the object belongs to one particular class.

[0070] In an effective embodiment, in step 8, the input image 1 is displayed on a display device such as a screen connected to, for example, an image collection unit, and a bounding box 5 having a sufficiently high confidence score is drawn. Then, in step 6, based on the detected anatomical structures 3, their spatial positions and / or relationships to each other, the probability of a given medical condition of the subject, such as a normal / abnormal condition (e.g., IUP pregnancy or non-IUP pregnancy), can be determined. Thus, the inference method 6 uses the bounding box 5 calculated by the object detection function 4 and includes algorithms that can calculate, for example, whether a bounding box 5 of a particular class is completely contained within another class, as well as the presence or absence of an anatomical structure 3 of a particular class. Also, the relative spatial position of the bounding box 5 is calculated and used to infer an appropriate probability 7 regarding the medical condition.

[0071] Figure 3 shows the object detection function 4 in more detail. The input is again one or several medical images 1 of the FOV2, at least some of which represent organs, i.e., anatomical structures 3. In one embodiment, the received medical images may be of any size and pixel size, but the first NN 40 works best when the image is M*2 Z ×M*2 ZThis is when the image is a square of size M, and M is an odd number. Therefore, in step 39, the medical image 1 is optionally upsampled or downsampled to fit the expected input size of the first neural network 40. The output of the first NN 40 is the coordinates of at least the bounding boxes 50, 51, and 52 (usually including confidence scores as well). If one of the detected bounding boxes has a class belonging to a predetermined large anatomical structure 3a, the detected bounding box 50 is used in step 42 to generate a cropped image 11 by cropping the medical image 1 to the first bounding box 50. "Cropping" means the operation performed, for example, by a snipping tool in photoprocessing, i.e., cutting out a smaller image 11 from a larger image 1 so that its edges align with the edges of the bounding box 50. Therefore, the cropped image 11 does not necessarily have to be a square image. Therefore, this image is preferably downsampled or upsampled in step 45, and as a result the cropped image 11 has preferably a predetermined size (e.g., a square 2D image for the first NN), and is then fed to the second NN 44. This second neural network then outputs at least one second bounding box 54 containing small-sized anatomical structures 3b. Small-sized anatomical structures 3b are generally very small relative to the FOV 2 or the entire organ or tissue imaged, or they have such fine tissue, so it is difficult to train the first NN 40 to detect them. However, when the image 1 is first cropped around the bounding box 50, it is not difficult to train the second NN 44 to detect this second class of small-sized anatomical structures 3b (and possibly only) using knowledge of the expected location of such tissues.

[0072] Figure 4 is a schematic diagram of one embodiment of inference method 6 that uses bounding boxes calculated from the first NN and the second NN. In the example of a first-term ultrasound scan, inference step 60 calculates whether a bounding box for the yolk sac (GS) is present. If present, the method proceeds to step 61. If not present, the probability of an abnormal pregnancy (medical condition 7c) is increased. In step 61, the algorithm determines whether the bounding box for the GS is a subset of the bounding box for the uterus. If yes, the probability of a normal IUP (condition 7a) is increased, and the method proceeds to step 62. Otherwise, i.e., the presence of the GS is not in the uterus, the probability of medical condition 7b "ectopic pregnancy" is increased. These steps are performed after the image has passed through the first NN 40 and before the trimming step 42 and the second NN 44 are applied. The second inference stage then detects and locates the yolk sac and embryo. Step 62 determines whether the bounding boxes for the yolk sac and / or embryo are a subset of the bounding box for the GS. If the answer is yes, the probability of a normal pregnancy (7a) increases. If no YS or embryo is detected inside the GS, the possibility of an abnormal pregnancy (7c) increases.

[0073] Figure 5 shows possible results of the hierarchical object detection function. Figure 5a shows identified bounding boxes around the uterus, GS, and embryo. By one embodiment of the present invention, the image is cropped around the GS bounding box, as shown in Figure 5b. A second NN is trained to detect YS within the GS, and the resulting bounding box is drawn in the figure.

[0074] Figure 6 shows a schematic diagram of the first and / or second NN, which is preferably a modification of the YOLOv3 small network. In this representation, each image input or feature map 20, 22 is annotated with its size (square image) at the top and the number of channels at the bottom. Thus, the input dataset 20 is a color image such as RGB, and is a square 2D image with a size of 416 × 416 pixels and 3 channels. In grayscale images, each channel generally has the same value. On the other hand, the layer immediately preceding the output layer 32a is only 13 × 13 in size and has a depth of 512 channels.

[0075] The input layer 20 is submitted to a 3x3 convolutional filter 24 with a stride of 1, followed by a 2x2 maxpool filter 28 with a stride of 2. More precisely, this layer uses 16 such convolutional filters 24, each with a depth of 3, to produce a feature map 22a with a size of 208 and a depth of 16, which is reduced to half the size of the input layer 20. The feature map 22a is convolved using another 3x3 convolutional filter with a stride of 1, followed by a 2x2 maxpool filter 28 with a stride of 2, resulting in a feature map 22b. This operation or block of layers, i.e., a 3x3 convolutional filter 24 with a stride of 1 followed by a 2x2 maxpool filter 28 with a stride of 2, is repeated two more times, resulting in a total of five convolutional layers 24 and each followed by a pool layer 28, each reduced to half the size in each iteration. Next, feature map 22e is again submitted to convolutional filter 24, but this time followed by maxpool 29 with size 2×2 and stride 1, and therefore no further reduction in size, resulting in the next feature map 22f having a depth of 512 and size 13×13. This layer is followed by another convolutional filter 24 with size 3×3 and stride 1, resulting in an output volume of 22g. This is submitted to convolutional filter 25 with size 1×1 and stride 1, which is used to reduce the depth from 1024 to 256. Therefore, convolutional filter 25 is sometimes called a feature map pooling or projection layer. This filter reduces the number of feature maps (number of channels) while preserving prominent features. The output 22h of this projection layer is submitted to another convolutional filter 24 with size 3x3 and stride 1, resulting in an output volume 22i. Finally, the convolutional filter 24 is followed by a convolutional filter 26 with size k and stride 1, where k = (C + 5) × B, where C is the number of classes and B is the number of anchor boxes, which is 3 in a preferred example.The resulting output layer 32a is referred to as YOLO inference at scale 1 and has the output format described above, namely containing bounding box data up to B (preferably 3) for each of the 13 × 13 grid points, where each bounding box contains four box coordinates, an object likelihood score, and individual class probabilities. The bounding boxes are filtered using thresholds relating to object likelihood and / or class scores.

[0076] To perform detection at scale 2, the early feature map 22d is passed through a convolutional filter 24 to yield feature map 22j. Furthermore, feature map 22h is submitted to a convolutional filter 25, followed by upsampling of size 2 with a stride of 1, yielding feature map 22l. This is concatenated with feature map 22j to yield feature map 22m. This is submitted to another 3x3 convolutional filter 24 to yield feature map 22n. This feature map is again submitted to a convolutional filter 26 to yield output volume (3D tensor) 32b, which thus contains the coordinates and probabilities of the bounding box of B in each cell of a higher-resolution 26x26 grid. The bounding box prediction at scale 1 and the bounding box prediction at scale 2 are combined as described above.

[0077] Figure 7 schematically illustrates the training method. Figure 7 shows the training images 12. For the example described herein, ultrasound scans of fetuses less than 11 weeks gestation were collected for algorithm development. Each image frame collected from the ultrasound scan was then used for manual annotation. Each image was annotated with axis-aligned bounding boxes covering the entire anatomical structure (one example being the U, GS, E, and YS shown in Figure 1). Data distribution was ensured to be uniform, with equal weighting given to all possible gestational ages. For example, for the GS, U, and embryo, 500–5,000 images 12 were annotated by drawing bounding boxes, and each annotation is shown in 70, 71, and 72 of Figure 7. The annotation 70 for the gestational sac is used to crop the gestational sac (step 73). In the cropped image, the yolk sac is annotated, and the annotation is saved as 75.

[0078] Thus, the training image 12 is used as input training data, and the GS annotation 70, uterus annotation 71, and embryo annotation 72 are used as output training data to train the first neural network in step 76. Therefore, the image 73 trimmed around the GS and the yolk sac annotation 75 are used to train the second neural network in step 78.

[0079] Figure 8 is a schematic diagram of an ultrasound system 100 according to one embodiment of the present invention, configured to perform the method of the present invention. The ultrasound hardware unit 102 included in the ultrasound system 100 includes a CPU 104, a GPU 106, and a digital storage medium 108, such as a hard disk or solid state disk 2019PF00350 20. A computer program is loaded into the hardware unit from a CD-ROM 110 or via the internet 112. The hardware unit 102 is connected to a user interface 114, which includes a keyboard 116 and an optional touchpad 118. The touchpad 118 also functions as a display device for displaying imaging parameters. The hardware unit 102 is connected to an ultrasound probe 120, which includes an array of ultrasound transducers 122, enabling the acquisition of B-mode ultrasound images from a subject or patient (not shown), preferably in real time. The B-mode image 124 acquired using the ultrasonic probe 120, and the bounding box 5 generated by the CPU 104 and / or GPU performing the method of the present invention, are displayed on a screen 126, which is any commercially available display unit, such as a screen, television set, flat screen, or projector. Furthermore, a remote computer or server 128 is connected, for example, via the internet 112. The method according to the present invention is performed by the CPU 104 or GPU 106 of the hardware unit 102, but may also be performed by the processor of the remote server 128.

[0080] The above discussion is intended to be merely illustrative of the System and should not be interpreted as limiting any attached claims to a particular embodiment or group of embodiments. Therefore, although the System is described in particular detail with reference to illustrative embodiments, it should be understood that many modifications and alternative embodiments can be devised by those skilled in the art without departing from the broadly intended purpose and scope of the System as expressed in the following claims. Accordingly, the specification and drawings should be considered illustrative and not intended to limit the scope of the attached claims. [Explanation of symbols]

[0081] Medical images FOV anatomical structure 3a Large-sized anatomical structures 3b Small-sized anatomical structures Object detection function Boundary box Inference method Medical conditions and their probability 7a Medical conditions and their probability 7b Medical conditions and their probability 7c Medical conditions and their probability 8 Display 11. Cropped image 12 training images 20 Input Layers 22 Feature Map 24. A convolutional filter with dimensions of 3x3 and a stride of 1. 25. A convolutional filter with a size of 1x1 and a stride of 1. 26. A convolutional filter with size K = (C + 5) × B and stride 1. Maxpool, size 28, 2x2, with a stride of 2. Maxpool, size 28, 2x2, stride 1. 30 size 2x2 with stride 1 upsampling 32a output layer 39. Upsampling or Downsampling 40. The First Neural Network 42 Trimming Steps 44. The Second Neural Network 45 Upsampling or Downsampling 50 The first bounding box (coordinates) 51 Bounding box (coordinates) 52 Bounding box (coordinates) 54. Second bounding box (coordinates) 60 IF / ELSE inference steps 61 IF / ELSE Inference Step 62 IF / ELSE Inference Steps 70 GS Note 71 U Note 72 Embryo annotation 73. Trimming the gestational sac 75 YS annotation 76 Training Step NN1 78 Training Step NN2 100 Ultrasonic Systems 102 Hardware Units 104 CPU 106 GPU 108 Digital storage media 110 CD-ROM 112 Internet 114 User Interface 116-key keyboard 118 Touchpads 120 Ultrasound Probes 122 Ultrasonic Transducer 124 B-mode images 126 screens 128 Remote Servers

Claims

1. A computer-based method for automatically detecting anatomical structures in medical images of a subject, wherein the computer-based method is: a) The step of receiving at least one medical image of the subject's FOV, b) A computer implementation method comprising the steps of applying an object detection function to the medical image to generate the coordinates of a plurality of bounding boxes and a confidence score for each bounding box as output, wherein the object detection function is trained to detect multiple classes of anatomical structures, and the confidence score indicates the probability that the bounding box contains an anatomical structure belonging to one of the multiple classes, wherein the object detection function The steps include: applying a first neural network trained to detect large-sized anatomical structures of a first class to the medical image, thereby generating as output the coordinates of at least one first bounding box and a confidence score indicating that the first bounding box contains a large-sized anatomical structure; The steps include: cropping the medical image to the first bounding box to generate a cropped medical image that includes the image content within the first bounding box; The steps include: applying a second neural network trained to detect at least one small-sized anatomical structure of a second class to the trimmed medical image, thereby generating as output the coordinates of at least one second bounding box and the confidence score that the second bounding box contains a small-sized anatomical structure; Execute, The object detection function is based on a hierarchical relationship between large-sized and small-sized anatomical structures, and it is expected that at least one of the first class of large-sized anatomical structures contains one or more of the class of small-sized anatomical structures. A computer implementation method characterized by the following features.

2. c) The computer-aided method according to claim 1, further comprising the step of determining the probability of a predetermined medical condition of the subject, wherein the probability of the predetermined medical condition is determined using an inference scheme based on the presence or absence of one or more classes of anatomical structures and / or the relative spatial position of the detected bounding box containing the anatomical structures.

3. The computer-assisted method according to claim 1 or 2, wherein the probability of a predetermined medical condition is increased if a first detected bounding box containing a first class of anatomical structures includes a second detected bounding box containing a second class of anatomical structures.

4. The computer implementation method according to any one of claims 1 to 3, wherein the computer implementation method is repeated for multiple two-dimensional medical images at different FOVs acquired during the same examination period of the subject, and the confidence score of the detected bounding box is used to calculate the optimal medical image or FOV for further evaluation.

5. The computer implementation method according to any one of claims 1 to 4, wherein the first neural network and / or the second neural network is a complete convolutional neural network.

6. The computer-aided method according to any one of claims 1 to 5, wherein the first neural network and / or the second neural network detect anatomical structures at two different scales, each scale being given by predetermined downsampling of the medical image.

7. The computer implementation according to any one of claims 1 to 6, wherein the first neural network and / or the second neural network is a complete convolutional neural network of YOLOv3.

8. The computer implementation method according to any one of claims 1 to 7, wherein the object detection function is trained to detect 2 to 12 anatomical structures.

9. The computer-aided method according to any one of claims 1 to 8, wherein the medical image is acquired during an ultrasound scan in the first trimester before childbirth, and the plurality of classes of anatomical structures include the uterus, gestational sac, embryo and / or yolk sac.

10. The computer-aided method according to claim 2 or 9, wherein the probability of a medical condition of "normal pregnancy" is increased if the detected boundary box of the uterus includes the detected boundary box of the gestational sac, and the boundary box of the detected gestational sac includes the detected boundary box of the embryo and / or yolk sac.

11. A method for training an object detection function for detecting multiple classes of anatomical structures in medical images, wherein the object detection function includes a first neural network, and the method is (a) The step of receiving input training data, i.e., at least one medical image of the subject's FOV, (b) Steps of receiving output training data, i.e., a tensor comprising the coordinates of at least one first bounding box in the medical image containing a large-sized anatomical structure belonging to one of a first class of large-sized anatomical structures, and a number indicating the large-sized anatomical structure of the class, (c) A step of training the first neural network using the input training data and the output training data. (d) The step of receiving a cropped image which includes input training data, i.e., the image content of the first bounding box containing a large-sized anatomical structure, (e) The step of receiving output training data, i.e., a tensor having the coordinates of at least one second bounding box inside the trimmed image, which includes small-sized anatomical structures belonging to at least one second class of small-sized anatomical structures, (f) A step of training a second neural network using the input training data and the output training data. Methods that include...

12. The training method according to claim 11, wherein the output training data includes a tensor having size N × N × [B * (4 + 1 + C)], where N × N is the size of the final feature map, B is the number of anchor boxes, C is the number of classes, and the number of anchor boxes is 3 or 6.

13. The training method according to claim 11 or 12, wherein the output training data is generated by applying a 1x1 detection kernel to a downsampled feature map, the shape of the detection kernel is 1x1x(B*(5+C)), where B is the number of anchor boxes, C is the number of classes, and the number of anchor boxes is 3 or 6.

14. A computer program including instructions, wherein a computing unit executes the computer program to perform the method according to any one of claims 1 to 13.

15. A system for automatically detecting anatomical structures in medical images of a subject, wherein the system is a) A first interface configured to receive at least one medical image of the subject's FOV, b) A computing unit configured to apply an object detection function to the medical image and thereby generate the coordinates of a plurality of bounding boxes and a confidence score for each bounding box as output, wherein the object detection function is trained to detect multiple classes of anatomical structures, the confidence score indicates the probability that the bounding box contains an anatomical structure belonging to one of the multiple classes, and the computing unit, The steps include: applying a first neural network trained to detect large-sized anatomical structures of a first class to the medical image, thereby generating as output the coordinates of at least one first bounding box and a confidence score indicating that the first bounding box contains a large-sized anatomical structure; The steps include: generating a cropped image that includes the image content within the first bounding box by cropping the medical image into the first bounding box; The steps include: applying a second neural network trained to detect at least one small-sized anatomical structure of a second class to the cropped medical image, thereby generating as output the coordinates of at least one second bounding box and the confidence score that the second bounding box contains a small-sized anatomical structure; Execute, The object detection function is based on a hierarchical relationship between large-sized and small-sized anatomical structures, and it is expected that at least one of the first class of large-sized anatomical structures contains one or more of the class of small-sized anatomical structures. system.

Images