System and method for detecting laterality of medical images

Abstract

The present invention is entitled a system and method for detecting the laterality of a medical image. The present invention provides an x-ray image laterality detection system (200). The x-ray image laterality detection system includes a detection computing device (202, 800). A processor (814) of the computing device is programmed to execute a neural network model for analyzing x-ray images, wherein the neural network model (204, 338, 600, 1600, 2600, 3600) is trained with a training x-ray image as input and with an observed laterality category associated with the training x-ray image as output. The process is also programmed to receive an unclassified x-ray image, analyze the unclassified x-ray image using the neural network model, and assign a laterality category to the unclassified x-ray image. If the assigned laterality category is not a target laterality, the processor is programmed to adjust the unclassified x-ray image to derive a corrected x-ray image with a target laterality and output the corrected x-ray image. If the assigned laterality category is a target laterality, the processor is programmed to output the unclassified x-ray image.

Description

Background Art

[0001] The field of the present disclosure generally relates to systems and methods for detecting laterality of images, and more specifically, to systems and methods for detecting laterality of medical images using a neural network model.

[0002] Medical images are typically stored and transmitted via a Picture Archiving and Communication System (PACS) according to the Digital Imaging and Communications in Medicine (DICOM) standard. According to the DICOM standard, in addition to image data, medical images also include metadata. Metadata is data associated with an image, such as information related to the patient, image acquisition, and imaging equipment. For example, metadata includes the patient's laterality, which indicates which side of the patient is shown on the left side of the image. Metadata is typically generated based on the imaging protocol used to acquire the medical image. The DICOM standard allows the communication and management of medical images and the integration of medical devices such as scanners, workstations, and PACS viewers across different manufacturers.

[0003] Due to its mobility and relatively small size, portable x-ray imaging has become one of the most common imaging modes in the field of medical imaging. Due to human error, the laterality of images acquired by portable x-ray imaging may be incorrect. Medical images with upright laterality are displayed as the right side of the image showing the left side of the patient. However, the operator of the device may incorrectly place the detector in the wrong way, such as placing the detector in front of the patient when taking a chest x-ray image of the anterior-posterior (AP) view, or enter the wrong laterality of the image in the user interface. Therefore, a flipped image is generated. Although the physician has the medical skills to determine that the generated image reflects a laterality different from the laterality stored in the image metadata, the incorrect laterality may still cause the physician to spend extra time to determine the correct laterality of the image before performing a diagnosis. In addition, having incorrect laterality information in the metadata can cause problems with the suspension protocol, which is used to display the image in a way that the user feels more useful when sending the image to PACS. Manually rotating the image before sending the image to PACS and storing the correct laterality in the metadata is an inefficient use of the technician's time. Furthermore, if images with incorrect laterality are used as input, they may degrade the performance of computer-aided diagnosis systems or artificial intelligence diagnosis systems. Summary of the invention

[0004] In one aspect, an x-ray image laterality detection system is provided. The x-ray image laterality detection system includes a detection computing device, which includes at least one processor in communication with at least one memory device. At least one processor is programmed to execute a neural network model for analyzing x-ray images, wherein the neural network model is trained with a training x-ray image as input and with an observed laterality category associated with the training x-ray image as output. At least one processor is also programmed to receive an unclassified x-ray image, analyze the unclassified x-ray image using the neural network model, and assign a laterality category to the unclassified x-ray image based on the analysis. If the assigned laterality category is not a target laterality, at least one processor is programmed to adjust the unclassified x-ray image to derive a corrected x-ray image with a target laterality and output the corrected x-ray image. If the assigned laterality category is a target laterality, at least one processor is programmed to output an unclassified x-ray image.

[0005] In another aspect, an image laterality detection system is provided. The image laterality detection system includes a detection computing device including at least one processor in communication with at least one memory device. The at least one processor is programmed to execute a neural network model for analyzing images, receive training images and observed laterality categories associated with the training images, and analyze the training images. The at least one processor is further programmed to determine a predicted laterality category for the training images using the neural network model, compare the predicted laterality category to the observed laterality category, and adjust the neural network model based on the comparison.

[0006] In yet another aspect, a method for detecting laterality of a medical image is provided. The method includes executing a neural network model for analyzing a medical image, wherein the neural network model is configured to detect laterality of the medical image. The method also includes receiving an unclassified medical image, analyzing the unclassified medical image using the neural network model, and detecting laterality of the unclassified medical image using the neural network model. The method also includes alerting a user whether the unclassified medical image has a target laterality based on the detected laterality. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1A is a schematic diagram of an apparatus for taking chest x-ray images.

[0008] Figure 1B is used Figure 1A An exemplary chest x-ray image of the device is shown.

[0009] Figure 1C is a schematic diagram of another apparatus for taking chest x-ray images.

[0010] Figure 1Dis used Figure 1C An exemplary chest x-ray image of the device is shown.

[0011] Figure 2A is a schematic diagram of an exemplary system for detecting laterality of a medical image.

[0012] Figure 2B According to one aspect of the present disclosure Figure 2A An exemplary sequence diagram of the flow of data in the illustrated system.

[0013] Figure 2C According to another aspect of the present disclosure Figure 2A An exemplary sequence diagram of the flow of data in the illustrated system.

[0014] Figure 3A According to another aspect of the present disclosure Figure 2A An exemplary sequence diagram of the flow of data in the illustrated system.

[0015] Figure 3B According to another aspect of the present disclosure Figure 2A Another exemplary sequence diagram of data flow in the illustrated system.

[0016] Figure 4 yes Figure 3B An exemplary sequence diagram of the data flow of the illustrated system.

[0017] Figure 5A It is a schematic diagram of the neural network model.

[0018] Figure 5B yes Figure 5A Schematic diagram of neurons in the neural network model shown.

[0019] Fig. 6A is a block diagram of an exemplary convolutional neural network.

[0020] Figure 6B is a block diagram of another exemplary convolutional neural network.

[0021] Figure 6C is a block diagram of yet another exemplary convolutional neural network.

[0022] Fig.6D is a block diagram of yet another exemplary convolutional neural network.

[0023] Fig. 7A is a flow chart of an exemplary method of detecting laterality of a medical image.

[0024] Figure 7B is a flow chart of another exemplary method of detecting laterality of a medical image.

[0025] Figure 8is a block diagram of an exemplary computing device. DETAILED DESCRIPTION

[0026] The present disclosure includes systems and methods for detecting the laterality of an image using a neural network model. Laterality as used herein refers to whether a medical image is upright to represent the left side of a patient or whether the medical image is displayed as if flipped. Flipped can be flipped horizontally, flipped vertically, or flipped along an oblique axis. Figure 1B and Figure 1D The chest x-ray image in is used as an exemplary image to illustrate the systems and methods described herein. However, the applicability of the systems and methods is not limited to chest x-ray images. The systems and methods are also applicable to x-ray images of other anatomical structures or mammographic images, and to other types of medical images, such as magnetic resonance imaging (MRI) images, computed tomography (CT) images, positron emission tomography (PET) images, and ultrasound images. Methodological aspects will be discussed in part as apparent and in part as explicitly in the following description.

[0027] Chest examinations performed with portable x-ray systems are one of the most frequently performed procedures. The acquired images are typically in an anterior-posterior rendering view, where the images are taken as if the camera is aimed at the patient from the front of the patient towards the back of the patient.

[0028] Figures 1A to 1D Exemplary laterality categories are shown. To take a chest x-ray image using a protocol for an AP view, the detector is placed behind the patient and the x-ray source 101 is placed in front of the patient (eg, Figure 1A shown). Figure 1B An x-ray image 102 acquired with such positioning of the detector is shown. Metadata for the x-ray image 102 indicates that the right side of the image shows the left side of the patient. Pending protocols in PACS are typically based on DICOM metadata. Thus, the patient's heart 104 is correctly shown in the x-ray image 102 as being positioned on the patient's left side (e.g., Figure 1B ). The x-ray image 102 has a positive laterality category. In contrast, when the technician places the detector in front of the patient and the x-ray source 101 behind the patient, the expected view is posterior-anterior (PA) (as shown). Figure 1C Instead, the technician incorrectly selected the protocol for the AP. Figure 1D Another acquired x-ray image 106 with this positioning of the detector is shown. Because the metadata for the laterality of the x-ray image 106 indicates that the right side of the image shows the left side of the patient, the left side of the patient is shown on the left side of the x-ray image 106, such that the patient's heart 104 is incorrectly displayed as being on the right side of the patient (e.g., Figure 1D). That is, the x-ray image 106 is displayed as horizontally flipped or mirrored. The x-ray image 106 has a flipped laterality category. The two laterality categories are used as examples only. The specific implementation of the system and method disclosed herein is not limited to two laterality categories. For example, the laterality category may include more than two upright and flipped categories, and may also include horizontal flipping, vertical flipping, flipping along a tilt axis, or any combination thereof.

[0029] Traditionally, guide markers 108 are placed next to the patient's anatomy to indicate the left or right side of the patient. Guide markers require additional workflow steps and are prone to human error, such as using the wrong letter or omitting a letter due to inadvertence. Figure 1B ), the guide marker 108 of the letter L is correctly shown as being on the left side of the patient. However, in the x-ray image 106 (as shown Figure 1D ), the guide marker 108 is flipped and incorrectly shown as being on the patient's right side.

[0030] To assess the prevalence of mirrored chest x-ray images, an analysis of 7,057 clinical images was performed. The analysis showed that 18.65% of the images were mirrored. In many systems, technicians are required to manually rotate images to correct mirrored images before sending them to PACS. If technicians double-click to correct mirrored images, technicians may spend 6.74 hours per year on manual corrections. With an artificial intelligence (AI) algorithm in the systems and methods described herein that is 99.3% accurate, technicians may spend 15 minutes per year on manual corrections. In addition, guide markers can be completely eliminated, reducing human error and speeding up workflow.

[0031] Figure 2A 2 is a schematic diagram of an exemplary system 200 for laterality detection of an image. The image may be a medical image. The system 200 includes a laterality detection computing device 202 configured to detect and correct laterality of an input image. The computing device 202 also includes a neural network model 204.

[0032] In an exemplary embodiment, the system 200 also includes a metadata editor 206 configured to update metadata of the image. The system 200 may also include a user interface manager 208 configured to receive user input regarding selections when detecting and correcting the laterality of the input image based on the input. For example, the user may turn on or off correction of the laterality of the input image. Due to rare medical conditions such as dextrocardia and situs inversus (where a person's heart may reside on the right side of the chest), the user may not want the image to be automatically flipped when the detected laterality would be incorrect for normal anatomy.

[0033] The system 200 may also include a post-processor 210 for post-processing the image after the laterality of the image has been classified and / or the image has been corrected. Post-processing may include, but is not limited to, applying a mapping of intensity levels to enhance the image, thereby matching the preferences of the radiologist. In some embodiments, the system 200 is configured to detect whether the guide mark is flipped, and the result is compared with the detected laterality of the input image to determine whether the guide mark is flipped when the laterality of the image is upright, or whether the guide mark is flipped and displayed due to the laterality of the flipped image. Post-processing may also include post-processing associated with the guide mark and / or digital marker, such as preventing a wrongly placed or flipped guide mark, replacing a flipped mark, replacing an erroneous digital marker, or generating and displaying a digital marker.

[0034] Figure 2B An exemplary sequence diagram 250 of data flowing through the system 200 without user input from the user interface manager 208 is shown. In the sequence 250, the unclassified image 209 is input into the laterality detection computing device 202. The output from the computing device 202 is input into the post-processor 210 and the metadata editor 206. The output from the post-processor 210 and the metadata editor 206 is then input into the user interface 212 to display the output image from the post-processor 210 with updated metadata 216 for the image. The output image can be a corrected image of the unclassified image 209. Alternatively, if the user has turned off correction for laterality, the output image can be a post-processed image of the unclassified image 209 without correction. An alert indicating the laterality of the displayed image can be provided on the image display 214 or the user interface 212. The alert can indicate whether the laterality is upright or inverted. The alert can also indicate upright placement of the marker. In one example, the alert may indicate whether the detected laterality matches the protocol used, the guide markers placed, the digital markers placed, or any combination thereof. In another example, the alert may indicate that the laterality of the unclassified image 209 is flipped and the laterality of the displayed image has been adjusted or corrected. In another example, the alert may indicate whether the guide markers or digital markers are placed upright. Output from the post-processor 210 and metadata editor 206 may also be sent to the PACS for archiving and transmission to other systems for further processing.

[0035] Figure 2CAnother exemplary sequence diagram 260 of data flowing through the system 200 is shown, wherein user input is provided from the user interface manager 208. The user interface manager 208 receives a user policy including input from a user. The user may choose to detect the laterality category of the input image without correcting its laterality. The user may also choose to both detect the laterality category and correct the laterality of the input image. In some embodiments, the user may select the timing of performing the functions of the laterality detection computing device 202. For example, when a chest x-ray is to be acquired, the user may choose not to correct for laterality.

[0036] In an exemplary embodiment, the user interface manager 208 communicates with the computing device 202, the metadata editor 206, and the post-processor 210 to transmit user input and update the display on the user interface. In sequence 260, the unclassified image 209 is provided to the laterality detection computing device 202. The output of the computing device 202 is provided to the post-processor 210 and the user interface manager 208. The output image 218 is output by the post-processor 210. The input image metadata 220 is provided to the metadata editor 206. The output image metadata 222 is output from the metadata editor 206. Compared to the sequence 250, in the sequence 260, where the user policy regarding the process of detecting and correcting the laterality is predefined, the user policy is provided by the user interface manager 208.

[0037] Figure 3A An exemplary sequence diagram 320 (in Figure 2A ). An unclassified image 209 that has not yet been classified for its laterality category is input into the neural network model 204. The unclassified image 209 may have been preprocessed, such as being resized to derive an image of the resized size. In some embodiments, the unclassified image 209 may not have been preprocessed. The neural network model 204 is configured to output the laterality category of the unclassified image 209. In an exemplary embodiment, the neural network model 204 provides one or more outputs 333 including the laterality category of the unclassified image 209. The computing device 202 may also include a corrector 324 configured to correct the laterality of the unclassified image 22. If the laterality correction is turned on based on the laterality category output by the neural network model 204, the unclassified image 209 is adjusted to have the laterality of the target laterality, such as being upright in the case where the left side of the patient is shown on the right side of the image. Therefore, an output image 218 with the target laterality is output from the laterality detection computing device 202. In some embodiments, when laterality correction is turned off, the laterality of the output image 218 is not adjusted even if the laterality category detected by the neural network model 204 is not the target laterality.

[0038] In an exemplary embodiment, to train the neural network model 204, training x-ray images are provided as inputs to the neural network model, and observed laterality classes associated with the training x-ray images are provided as outputs of the neural network model 204. The observed laterality classes may include orthogonal (see Figure 1B ) and flipped (see Figure 1D ).

[0039] The training x-ray images may be preprocessed before being provided to the neural network model. Exemplary preprocessing algorithms include, but are not limited to, lookup table mapping, histogram equalization, normalization, intensity transformation, gradient calculation, edge detection, or a combination thereof. The training x-ray images may be reduced in size before being provided to the neural network model to reduce the computational burden of the neural network model on the computing device. The training x-ray images may be reduced in size by reducing the image resolution of the training x-ray images to generate reduced-size training x-ray images. In one example, the unclassified image 209 may have been applied with these preprocessings before being input into the laterality detection computing device 202.

[0040] In some embodiments, features of the training x-ray images may be extracted before being provided to the neural network model. Image features are typically derived from the distribution of intensity values ​​of the image pixels. For example, a histogram of orientation gradient (HOG) features is derived by analyzing the gradient laterality in a local area of ​​the image. The image is divided into small areas of different sizes (called cells). Adjacent cells can be combined in larger areas called blocks. HOG features are not invariant to laterality. Features may indicate edges in the training x-ray images or landmarks (such as certain anatomical structures of a patient). The features extracted from the training x-ray images are then provided to the neural network model. These features can be used in the supervised learning algorithm of the neural network model.

[0041] Figure 3B Another exemplary sequence diagram 350 of data flow in system 200 is shown. Figure 3A Unlike the neural network model shown, the neural network model 204 is configured to detect and correct the laterality of the input image. An unclassified image 209 is input to the neural network model 204. The neural network model 204 converts the unclassified image 209 directly into a corrected image 218, which has the target laterality if the laterality correction is turned on.

[0042] Figure 4An exemplary sequence diagram 400 of data flow in system 200 is shown, wherein computing device 202 is implemented with neural network model 204, which is configured to detect and correct the laterality of unclassified image 209. Unclassified image 209 is input into neural network model 204. Neural network model 204 directly converts unclassified image 209 into corrected image 218 having a target laterality, such as upright. Specifically, in neural network model 204, unclassified image 209 is flipped to derive flipped image 334. Multiple images 335 including flipped image 334 and unclassified image 209 are input into convolutional neural network 338 to predict which of the images has the target laterality. Convolutional neural network 338 predicts an index associated with the image having the target laterality. A selector layer or operation 340 in neural network model 204 selects and outputs image 218 associated with the predicted index.

[0043] In an exemplary embodiment, the neural network model 204 is trained by inputting an image with a known (real) laterality into the neural network model 204. Within the neural network model 204, multiple images are generated including an input image and a flipped image of the input image. Flipping is an operation and does not have trainable parameters. A live vector is generated based on the reality of which image between the input image and the flipped image has the target laterality. For example, a live vector of

[10] indicates that the input image has the target laterality, and a live vector of

[01] indicates that the flipped image has the target laterality. The flipped image and the input image are provided to the convolutional neural network 338 together with the live vector to train the "selector" model, i.e., the convolutional neural network 338. The convolutional neural network 338 predicts an index associated with an image having the target laterality. The weights of the convolutional neural network 338 are updated based on the predicted index compared to the live vector. The selector layer or operation selects and outputs an image associated with the predicted index.

[0044] although Figure 4 , but the laterality category may also include horizontal flipping, vertical flipping, flipping along an oblique axis, or any combination thereof. The image 335 input to the neural network model 338 may include multiple flipped images 334 flipped along various axes.

[0045] Figure 5A An exemplary artificial neural network model 204 is shown. The exemplary neural network model 204 includes neuron layers 502, 504-1 to 504-n, and 506, including an input layer 502, one or more hidden layers 504-1 to 504-n, and an output layer 506. Each layer may include any number of neurons, i.e., Figure 5A Where q, r and n can be any positive integer. It should be understood that Figure 5AThe structures and configurations shown are different structures and configurations of neural networks to implement the methods and systems described herein.

[0046] In an exemplary embodiment, the input layer 502 may receive different input data. For example, the input layer 502 includes a first input a1 representing a training x-ray image, a second input a2 representing a pattern identified in the training x-ray image, a third input a3 representing an edge of the training x-ray image, etc. The input layer 502 may include thousands or more inputs. In some embodiments, the number of elements used by the neural network model 204 changes during the training process, and if, for example, during the execution of the neural network, it is determined that some neurons have less relevance, then these neurons are bypassed or ignored.

[0047] In an exemplary embodiment, each neuron in the hidden layers 504-1 to 504-n processes one or more inputs from the input layer 502 and / or one or more outputs from a neuron in one of the previous hidden layers to generate a decision or output. The output layer 506 includes one or more outputs, each indicating a label, a confidence factor, a weight describing the input, and / or an output image. The confidence factor and / or weight reflect the strength of the laterality of the output laterality category indication image. However, in some embodiments, in addition to or instead of the outputs from the output layer 506, the outputs of the neural network model 204 are obtained from the hidden layers 504-1 to 504-n.

[0048] In some embodiments, each layer has a discrete, identifiable function with respect to the input data. For example, if n=3, the first layer analyzes the first dimension of the input, the second layer analyzes the second dimension of the input, and the last layer analyzes the third dimension of the input. The dimensions may correspond to aspects that are considered to be of strong certainty, then to those aspects that are considered to be of medium importance, and finally to those aspects that are considered to be less relevant.

[0049] In other embodiments, the layers are not clearly delineated in terms of the functions they perform.For example, two or more of the hidden layers 504-1 through 504-n may share decisions related to labels, with no single layer making independent decisions regarding labels.

[0050] Figure 5B According to one embodiment, the corresponding Figure 5A Example neuron 550 of a neuron labeled "1, 1" in hidden layer 504-1 of FIG. For each input to neuron 550 (e.g., Figure 5A The input layer 502 in the input layer is weighted so that the input a1 to a p corresponds to the weights w1 to w2 determined during the training process of the neural network model 204p .

[0051] In some embodiments, some inputs lack explicit weights, or have weights below a threshold. The weights are applied to a function α (labeled by reference numeral 510), which may be a summation and may produce a value z1 that is input to the input labeled f 1，1 (z1) is a function 520. Function 520 is any suitable linear or nonlinear function. Figure 5B As shown, function 520 generates multiple outputs, which can be provided to neurons in subsequent layers or used as outputs of neural network model 204. For example, the output can correspond to an index value of a label list, or can be a calculated value used as an input to a subsequent function.

[0052] It should be understood that the structure and function of the neural network model 204 and neurons 550 shown are for illustration purposes only, and other suitable configurations exist. For example, the output of any given neuron may depend not only on the values ​​determined by past neurons, but also on future neurons.

[0053] The neural network model 204 may include a convolutional neural network, a deep learning neural network, a reinforcement or reinforcement learning module or program, or a combined learning module or program that learns in two or more areas or aspects of interest. Deep learning networks show superior performance in terms of accuracy compared to non-deep learning networks. The neural network model 204 can be trained using supervised or unsupervised machine learning programs. Machine learning can involve identifying and recognizing patterns in existing data in order to make predictions about subsequent data. Models can be created based on exemplary inputs in order to make effective and reliable predictions about new inputs.

[0054] In addition or alternatively, a machine learning program may be trained by inputting sample data sets or certain data such as images and object statistics and information into the program. The machine learning program may utilize a deep learning algorithm that may focus primarily on pattern recognition and may be trained after processing multiple examples. The machine learning program may include Bayesian program learning (BPL), speech recognition and synthesis, image or object recognition, optical character recognition, and / or natural language processing, alone or in combination. The machine learning program may also include natural language processing, semantic analysis, automated reasoning, and / or machine learning.

[0055] Both supervised and unsupervised machine learning techniques may be used. In supervised machine learning, a processing element may be provided with exemplary inputs and their associated outputs, and may attempt to discover general rules that map inputs to outputs, so that when subsequent new inputs are provided, the processing element can accurately predict the correct output based on the discovered rules. In unsupervised machine learning, a processing element may need to find its own structure in unlabeled exemplary inputs.

[0056] Based on these analyses, the neural network model 204 can learn how to identify features and patterns that can then be applied to analyze image data, model data, and / or other data. For example, the model 204 can learn to identify the laterality of an input image.

[0057] Fig. 6A An exemplary convolutional neural network 600 according to one aspect of the present disclosure is shown. The neural network model 204 includes the convolutional neural network 600, such as the convolutional neural network 338 (e.g. Figure 4 ). The convolutional neural network 600 includes a convolutional layer 608. In the convolutional layer 608, convolution is used instead of the general matrix multiplication in the neural network model. The neural network 600 includes one or more convolutional layer blocks 602, a fully connected layer 604 in which neurons in the layer are connected to each neuron in the previous layer, and an output layer 606 that provides an output.

[0058] In an exemplary embodiment, the convolutional layer block 602 includes a convolutional layer 608 and a pooling layer 610. Each convolutional layer 608 is flexible in terms of its depth, such as the number of convolutional filters and the size of the convolutional filters. The pooling layer 610 is used to simplify the underlying calculations and reduce the dimensionality of the data by combining the outputs of the neuron clusters at the previous layer into a single neuron in the pooling layer 610. The convolutional layer block 602 may also include a normalization layer 612 between the convolutional layer 608 and the pooling layer 610. The normalization layer 612 is used to normalize the distribution within the training image batch and update the weights in the layer after normalization. The number of convolutional layer blocks 602 in the neural network 600 may depend on the image quality of the training x-ray images and the level of detail in the extracted features.

[0059] In operation, during training, training x-ray images and other data, such as extracted features of the training x-ray images, are input into one or more convolutional layer blocks 602. The observed laterality class and / or the corrected training x-ray image is provided as an output of the output layer 606. The neural network 600 is adjusted during training. Once the neural network 600 is trained, an input x-ray image is provided to the one or more convolutional layer blocks 602, and the output layer 606 provides an output that includes the laterality class and may also include a corrected x-ray image of the input x-ray image.

[0060] Convolutional neural network 600 may be implemented as convolutional neural network 1600 ( Figure 6B As shown), 2600( Figure 6C As shown), 3600( Fig.6D shown). Figure 6B An exemplary convolutional neural network 1600 including a plurality of convolutional layers 608 is shown. The neural network 1600 includes layers 1602-1628. Each of the layers 1602-1614 includes various combinations of a convolutional layer 608 with a normalization layer 612 and a pooling layer 610. For example, layer 1602 includes a convolutional layer 608 and a pooling layer 610. In addition to the convolutional layer 608 and the pooling layer 610, layers 1604, 1606, 1610, 1614 also include a normalization layer 612. Layers 1608, 1612 include convolutional layers 608 and normalization layers 612. In one example, layers 1616, 1618 include convolutional layers 608. Layer 1620 is a global average pooling layer 614. The global average pooling layer 614 reduces overfitting by reducing the number of parameters. The global average pooling layer 614 can also make the extracted features spatially correlated. The output of the global average pooling layer 614 is provided to layer 1624. Layers 1624, 1626 are fully connected layers 604. The final layer 1628 of the neural network 1600 is the output layer 606, which determines whether the input 1630 is a flipped image or an upright image, or has a laterality class of flipped or upright.

[0061] Figure 6C Yet another exemplary convolutional neural network 2600 is shown. The convolutional neural network 2600 includes one or more pairs 2602 of convolutional blocks 2604 and residual blocks 2606. The convolutional blocks 2604 include convolutional layers 608. In the residual blocks, a layer may provide its output to a previous layer that is not immediately preceding. The convolutional neural network 2600 also includes a convolutional layer 608 after the pair 2602.

[0062] Fig.6D Yet another exemplary convolutional neural network 3600 is shown. The convolutional neural network 3600 includes a plurality of layers: a convolutional layer 608, a normalization layer 612, a pooling layer 610, a further convolutional layer 608, and then a global average pooling layer 614. In the convolutional neural networks 1600, 2600, 3600, the parameters of the convolutional layer 608, the normalization layer 612, the pooling layer 610, and the global average pooling layer 614 may be different or remain the same at different layers.

[0063] Fig. 7AA flow chart of an exemplary method 700 for detecting laterality of an x-ray image is shown. The method 700 includes executing 702 a neural network model. The method 700 also includes receiving 704 a training x-ray image and an observed laterality category associated with the training x-ray image. The observed laterality category is the laterality category of the training x-ray image, such as upright or flipped. The training x-ray image and the observed laterality category are input into a neural network with the training x-ray image as input and the observed laterality category as output.

[0064] In an exemplary embodiment, the method 700 also includes analyzing 706 the training x-ray images. Furthermore, the method 700 includes calculating 708 a predicted laterality class of the training x-ray images using the neural network model. Furthermore, the method 700 includes comparing 710 the laterality of the corrected training x-ray images with the target laterality. The method 700 also includes adjusting 712 the neural network model based on the comparison. For example, parameters of the neural network model and the number of layers and neurons are adjusted based on the comparison.

[0065] In one example, instead of receiving an observed laterality class, an observed x-ray image associated with a training x-ray image is received. The observed x-ray image is the training x-ray image adjusted to have a target laterality, such as upright. The training x-ray image and the observed x-ray image are provided to a neural network model with the training x-ray image as input and the observed x-ray image as output. The neural network model predicts an x-ray image that corresponds to the training x-ray image and has the target laterality. The predicted x-ray image and the observed x-ray image are compared. The neural network model is adjusted based on the comparison.

[0066] Figure 7B A flow chart of another exemplary method 750 of detecting laterality of an input image is shown. The method 750 includes executing 752 a neural network model. The neural network model is trained with training x-ray images as input and with laterality classes associated with the training x-ray images as output.

[0067] In an exemplary embodiment, the method 750 also includes receiving 754 an unclassified x-ray image whose laterality has not been classified. The method 750 also includes analyzing 756 the unclassified x-ray image using a neural network model. A laterality category of the unclassified x-ray image is then assigned 758 based on the analysis. If the assigned laterality category is a target laterality, such as upright, the unclassified x-ray image has the correct laterality and is output 764. If the assigned laterality category is not the target laterality, the unclassified x-ray image is adjusted 760.

[0068] In some embodiments, the unclassified x-ray image is adjusted by flipping the unclassified x-ray image to have the laterality of the target laterality. If the assigned laterality category is being flipped, the unclassified image is flipped to the target laterality. In one example, if the assigned laterality category is being flipped, the guide mark can be digitally blocked or covered so that the guide mark does not confuse the reader. In another example, a new digital mark can be generated based on user-defined logic to indicate the correct laterality of the image. The digital mark can be the letter "L" or "R". The digital mark of the letter "L" can be placed on the right side of the medical image to indicate the left side of the patient. Alternatively, the digital mark of the letter "R" can be placed on the left side of the medical image to indicate the right side of the patient.

[0069] In one example, rather than using an observed laterality class associated with a training x-ray image, a neural network model is trained using the training x-ray image as input and an observed x-ray image associated with the training x-ray image as output, wherein the observed x-ray image has a target laterality. An unclassified x-ray image is conditioned using the neural network model, wherein the neural network model outputs a corrected x-ray image associated with the unclassified x-ray image.

[0070] The method 750 also includes outputting 762 the corrected x-ray image. In some embodiments, the method 750 includes using the neural network model to simultaneously output the corrected x-ray image and the laterality category of the input image. That is, the neural network model outputs both the corrected x-ray image and the laterality category of the input image.

[0071] In some embodiments, the unclassified image is not corrected even if the detected laterality category is flipping. Instead, after assigning 758 the laterality category, a warning is provided about the laterality of the unclassified image.

[0072] In an exemplary embodiment, metadata associated with the unclassified x-ray image may be updated based on the detected laterality class.Metadata associated with the output x-ray image is then generated to reflect the update.

[0073] The computing device 202, post-processor 210, user interface manager 208, metadata editor 206, and user interface 212 described herein may be implemented on any suitable computing device and software implemented therein. Figure 8800. In the exemplary embodiment, the computing device 800 includes a user interface 804 that receives at least one input from a user. The user interface 804 may include a keyboard 806 that enables the user to enter relevant information. The user interface 804 may also include, for example, a pointing device, a mouse, a stylus, a touch-sensitive panel (e.g., a touchpad and a touch screen), a gyroscope, an accelerometer, a position detector, and / or an audio input interface (e.g., including a microphone).

[0074] In addition, in this exemplary embodiment, the computing device 800 includes a presentation interface 807 that presents information such as input events and / or verification results to the user. The presentation interface 807 may also include a display adapter 808 coupled to at least one display device 810. More specifically, in this exemplary embodiment, the display device 810 may be a visual display device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED) display, and / or an "electronic ink" display. Alternatively, the presentation interface 807 may include an audio output device (e.g., an audio adapter and / or a speaker) and / or a printer.

[0075] The computing device 800 also includes a processor 814 and a memory device 818. The processor 814 is coupled to the user interface 804, the presentation interface 807, and the memory device 818 via a system bus 820. In this exemplary embodiment, the processor 814 communicates with the user, such as by prompting the user via the presentation interface 807 and / or by receiving user input via the user interface 804. The term "processor" generally refers to any programmable system, including system and microcontrollers, reduced instruction set computers (RISC), complex instruction set computers (CISC), application specific integrated circuits (ASICs), programmable logic circuits (PLCs), and any other circuits or processors capable of performing the functions described herein. The above examples are exemplary only and are therefore not intended to limit the definition and / or meaning of the term "controller" in any way.

[0076] In the exemplary embodiment, the memory device 818 includes one or more devices that enable information such as executable instructions and / or other data to be stored and retrieved. In addition, the memory device 818 includes one or more computer-readable media, such as but not limited to dynamic random access memory (DRAM), static random access memory (SRAM), solid-state disks, and / or hard disks. In the exemplary embodiment, the memory device 818 stores but is not limited to application source code, application object code, configuration data, additional input events, application status, assertion statements, verification results, and / or any other type of data. In the exemplary embodiment, the computing device 800 may also include a communication interface 830 connected to the processor 814 via the system bus 820. In addition, the communication interface 830 is communicatively connected to a data acquisition device.

[0077] In the exemplary embodiment, the processor 814 may be programmed by encoding operations using one or more executable instructions and providing the executable instructions in the memory device 818. In the exemplary embodiment, the processor 814 is programmed to select a plurality of measurements received from the data collection device.

[0078] In operation, a computer executes computer executable instructions embodied in one or more computer executable components stored on one or more computer readable media to implement various aspects of the present invention described and / or shown herein. Unless otherwise specified, the execution or implementation order of the operations in the embodiments of the present invention shown and described herein is not required. That is, unless otherwise specified, these operations can be performed in any order, and embodiments of the present invention may include more or fewer operations than those disclosed herein. For example, it is conceivable that performing or implementing a particular operation before, simultaneously with, or after another operation is within the scope of various aspects of the present invention.

[0079] At least one technical effect of the systems and methods described herein includes (a) automatic detection of laterality of x-ray images; (b) automatic adjustment of laterality of x-ray images; and (c) increased flexibility by providing a user interface manager to receive user input.

[0080] Exemplary embodiments of systems and methods for detecting and / or correcting laterality of medical images are described in detail above. These systems and methods are not limited to the specific embodiments described herein, but rather the components of the systems and / or operations of the methods may be used independently and separately from other components and / or operations described herein. In addition, the components and / or operations described may also be defined in other systems, methods, and / or devices, or used in conjunction with other systems, methods, and / or devices, and are not limited to practice with only the systems described herein.

[0081] Although specific features of various embodiments of the present invention may be shown in some drawings but not in other drawings, this is for convenience only. According to the principles of the present invention, any feature of a drawing may be referenced and / or claimed in combination with any feature of any other drawing.

[0082] This written description uses examples to disclose the invention, including the best mode, and also to enable those skilled in the art to practice the invention, including making and using any devices or systems and performing any included methods. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to fall within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with minor differences from the literal language of the claims.

Claims

1. An X-ray image laterality detection system (200), comprising: A detection computing device (202, 800) comprising at least one processor (814) in communication with at least one memory device (818), wherein the at least one processor is programmed to: receiving an unclassified x-ray image; executing a neural network model for analyzing x-ray images to analyze the unclassified x-ray image using the neural network model, wherein the neural network model is trained with training x-ray images as input and with observed laterality classes associated with the training x-ray images as output; assigning a laterality category to the unclassified x-ray image based on the analyzing; if the assigned laterality category is not a target laterality, the at least one processor is programmed to: adjusting the unclassified x-ray image to derive a corrected x-ray image having the target laterality; as well as outputting the corrected x-ray image; as well as If the assigned laterality category is the target laterality, the at least one processor is programmed to output the unclassified x-ray image, Wherein the target laterality comprises wherein the right side of the x-ray image shows an upright laterality of the patient's left side.

2. The system (200) of claim 1, wherein the unclassified x-ray image (209) is acquired without guide markers (108).

3. The system (200) of claim 1, wherein the neural network model (204, 338, 600, 1600, 2600, 3600) is a deep learning neural network model.

4. The system (200) of claim 1, wherein the neural network model (204, 338, 600, 1600, 2600, 3600) comprises a global average pooling layer.

5. The system (200) of claim 1, further comprising a metadata editor (206), the metadata editor being configured to: updating metadata associated with the unclassified x-ray image based on the detected laterality class of the unclassified x-ray image; and Metadata associated with the output x-ray image is generated.

6. The system (200) of claim 1, wherein the at least one processor (814) is further programmed to: generating a digital indicia indicative of the laterality of the output x-ray image; and The digital marker is overlaid on the output x-ray image.

7. The system (200) of claim 1, further comprising a user interface manager (208), the user interface manager being configured to: receiving user input; and generating a user policy based on the user input, Wherein the at least one processor (814) is further programmed to correct for laterality of the unclassified x-ray image based on the user policy.

8. The system (200) of claim 1, wherein the unclassified x-ray image (209) is pre-processed.

9. The system (200) of claim 1, wherein the at least one processor (814) is configured to alert a user that the unclassified x-ray image does not have the target laterality and that the output x-ray image is the corrected x-ray image.

10. The system (200) of claim 1, wherein the at least one processor (814) is configured to provide an alert indicative of the detected laterality of the unclassified x-ray image.

11. A method (700, 750) for detecting laterality of a medical image, the method comprising: receiving (754) an unclassified medical image; executing a neural network model for analyzing the medical image to analyze (756) the unclassified medical image using the neural network model, wherein the neural network model is configured to detect laterality of the medical image; detecting laterality of the unclassified medical image using the neural network model; as well as A user is alerted whether the unclassified medical image has a target laterality based on the detected laterality, wherein the target laterality comprises an upright laterality wherein the right side of the medical image shows a left side of the patient.

12. The method (700, 750) of claim 11, further comprising: The unclassified medical image is adjusted (760) to derive a corrected medical image.

13. The method (700, 750) of claim 12, wherein the neural network model (204, 338, 600, 1600, 2600, 3600) is trained with a training medical image as input and with an observed medical image associated with the training medical image as output, the observed medical image being the training medical image adjusted to have a target laterality, and adjusting (760) the unclassified medical image further comprises adjusting the unclassified medical image to have the target laterality using the neural network model.

14. The method (700, 750) of claim 11, wherein receiving (754) an unclassified medical image comprises receiving an unclassified x-ray medical image acquired without guidance markers.

15. The method (700, 750) of claim 11, wherein the neural network model (204, 338, 600, 1600, 2600, 3600) is trained with training medical images as input and with observed laterality classes associated with the training medical images as output, the method further comprising: assigning (758) a laterality category to the unclassified medical image based on the analysis; If the assigned laterality category is not the target laterality, the method further comprises: conditioning (760) the unclassified medical image to derive a corrected medical image having the target laterality; and outputting (762) the corrected medical image; and If the assigned laterality category is the target laterality, the method further includes (764) outputting the unclassified medical image.

Images