Methods and systems for managing medical image processing

Abstract

This invention provides a method and system for managing medical image processing. [Solution] The method includes: receiving a medical image 112 generated using a medical imaging device 111 from a user device 110 via a network by a processing server 150; performing image processing operations on the medical image based on image processing parameters; determining whether the image processing operations on the medical image have been completed by the processing server; and, in response to determining that the image processing operations on the medical image have been completed, sending a status message 183 to the user device via the network by the processing server.

Description

[[Technical Field]]

[0001] The present invention relates to a method and system for managing medical image processing. [[Background Art]]

[0002] Medical images have been generated in hospitals for many medical applications, such as diagnosis of a patient's symptoms and checking the progress of treatment. When a medical image is generated using a medical imaging technology such as computed tomography (CT) or magnetic resonance imaging (MRI), the medical image may comply with several medical information protocols before being uploaded to a cloud environment. Once uploaded, the medical image is not only shared among group hospitals and medical personnel, but also used for remote image diagnosis and data backup. [[Summary of the Invention]] [[Means for Solving the Problems]]

[0003] This summary is provided to introduce some of the concepts that will be described later in the detailed description of the invention. This summary is not intended to identify important features or essential features of the claimed subject matter, nor is it intended to be used as an aid in limiting the scope of the claimed subject matter.

[0004] Primarily, in one aspect, embodiments relate to a method comprising receiving, by a server, a medical image generated by a medical imaging apparatus from a user device via a network. The method further comprises performing an image processing operation on the medical image based on image processing parameters. The method further comprises determining, by the server, whether the image processing operation on the medical image is completed. The method further comprises transmitting, by the server, a status message to the user device via the network in response to determining that the image processing operation on the medical image is completed.

[0005] Primarily in one embodiment, the embodiment relates to a system including various user devices, a processing server, and a medical image management device coupled to the user devices and the processing server via a network. The medical image management device includes a computer processor. The medical image management device receives a medical image generated using a medical imaging device from one of several user devices via the network. The medical image management device determines whether or not the image processing operation on the medical image has been completed. The medical image management device sends a status message to the user device via the network in response to determining that the image processing operation on the medical image has been completed.

[0006] Primarily, in one embodiment, the embodiment relates to a non-temporary computer-readable medium containing instructions configured to perform a method when executed by a computer. The method includes receiving a medical image generated using a medical imaging device from a user device via a network. The method further includes determining whether an image processing operation on the medical image has been completed. The image processing operation is performed by a processing server. The method further includes sending a status message to the user device via the network in response to determining that an image processing operation on the medical image has been completed.

[0007] In light of the structure and function described above, embodiments of the present invention may include means adapted to perform the various processes and functions defined above, according to one or more embodiments and embodiments of the one or more embodiments described herein. Other aspects and advantages of the claimed subject matter will become apparent from the following description and the appended claims. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows a system according to several embodiments. [Figure 2] Figure 2 shows flowcharts for several embodiments. [Figure 3] Figure 3 shows examples of several embodiments. [Figure 4] Figure 4 shows examples of several embodiments. [Figure 5] Figure 5 shows examples of several embodiments. [Figure 6] Figure 6 shows examples of several embodiments. [Figure 7] Figure 7 shows examples of several embodiments. [Figure 8A] Figure 8A shows examples of several embodiments. [Figure 8B] Figure 8B shows examples of several embodiments. [Figure 9A] Figure 9A shows computing systems according to several embodiments. [Figure 9B] Figure 9B shows computing systems according to several embodiments. Detailed description of the invention

[0009] The following detailed description of embodiments of the Disclosure includes numerous specific descriptions to provide a better understanding of the Disclosure. However, it will be apparent to those skilled in the art that the Disclosure can be implemented without these specific descriptions. In other examples, known features are not described in detail to avoid unnecessarily complicating the description.

[0010] Throughout this application, ordinal numbers (e.g., first, second, third, etc.) may be used as adjectives for elements (i.e., any noun in this application). The use of ordinal numbers does not suggest or create a particular ordering of elements, nor does it limit elements to only a single element, unless explicitly disclosed, such as through the use of terms like "before," "after," or "single." Rather, the use of ordinal numbers is for distinguishing elements. For example, the first element is distinct from the second element, the first element encompasses multiple elements, and in the order of elements it succeeds (or precedes) the second element.

[0011] Generally, embodiments of this disclosure include systems and methods for managing image processing (IP) and / or artificial intelligence (AI) processing operations of medical images using a variety of status messages. For example, a medical image management device may provide network mediation between one or more processing servers that perform IP and AI processing operations and user devices that upload medical images to the cloud. In prior art systems, users are notified whether medical images have finished uploading to a particular network location, but they do not receive updates on the availability of medical images for further processing or use by healthcare professionals. With the advent of high-performance computing, some image and AI processing requires considerable computation time and computing resources to run within a cloud network. Thus, one or more medical image management devices can provide a smooth image diagnostic system by obtaining and reporting real-time information to the user regarding the progress of a particular process in the cloud (e.g., whether the processing operation has started, the amount of progress completed, and whether the processing operation has ended due to one or more processing errors).

[0012] Figure 1 shows a schematic diagram of a medical data network (e.g., medical data network X(100)) which may include various user devices (e.g., user device A(110), user device B(120), user device C(130)), various servers (e.g., server Z(150)), various network elements (not shown), and medical image management devices (e.g., medical image management device Y(170)). User devices include personal computers, smartphones, smartwatches, human-machine interfaces, hospital terminals (e.g., workstations that allow healthcare professionals to access the hospital's PACS (Picture Archiving and Communication System)), and other devices connected to a computer network that obtain user input from users. User devices may include hardware and / or software that has the functionality to provide input devices, display devices (e.g., display device B(121), display device C(131)), communication interfaces, and a graphical user interface (GUI). Network elements may refer to various hardware components within a computer network, such as switches, routers, and hubs, or logical entities for integrating one or more physical devices on a computer network such as the Internet, such as user devices, servers, network storage devices, and user equipment. Medical image management devices, user devices, processing servers, and network elements may include computing systems similar to the computing system (900) described in Figures 9A and 9B.

[0013] A user device may be connected to one or more medical imaging devices (e.g., medical imaging device A(111), medical imaging device B(124)) for generating medical images (e.g., medical image A(112), medical image data Y(175), medical image P(177), medical image Q(178)). The medical imaging devices may be X-ray machines, ultrasound machines, magnetic resonance imaging (MRI), positron emission tomography (PET), computed tomography (CT) scan machines, or image processing workstations. For example, an ultrasound machine includes hardware and / or software for emitting acoustic waves onto a biological object and detecting the reflected acoustic waves to generate an ultrasound image. These reflected acoustic waves may be analyzed to identify various properties (such as tissue density) of the subject's tissue through which one or more acoustic waves have passed. Thus, an ultrasound machine may include processing circuits, input / output devices, ultrasound circuits, and / or memory circuits, as well as other components such as transducer elements, waveform generators, and beamformers. On the other hand, an X-ray imaging system may include a radiation source, a radiation emission control device, a radiation detector, and a reading control device for viewing the X-ray image generated using the radiation detector. The radiation source may include an X-ray tube that accelerates high-energy electrons and directs them towards a target. Furthermore, the radiation detector may include a semiconductor image sensor such as a flat panel detector (FPD) for generating the X-ray image. In the case of an MRI system, the MRI system may include not only a powerful magnet but also a support structure (such as vertical or horizontal walls). Medical imaging systems also include multimodality systems such as MRI-CT systems and PET-MRI systems. An image processing workstation may include a system that receives images from one or more modalities, verifies the images, or performs image processing (such as image reconstruction), and then transmits the images to a medical image management system.

[0014] In response to the generation of medical images, a medical image file may be generated that associates the medical image with non-image content. For example, a medical image file may contain various data fields such as supplementary information including patient information, examination information, research information, series information, image-related information, and image ID. Supplementary information may be written to the header of the image file, or inserted as metadata in one or more embedded data fields. Series information includes information about a specific modality series (CT, MRI, DR, CR, US, etc.), a series instance UID, examination portion, series number, and series description. The series instance UID may be a unique identifier (UID) for identifying a specific series. The series number is a number indicating the order of the series in the same examination and may be used as the display order. Image-related information includes the image number, image generation date, imaging direction, imaging start position, image type, and file path. The image number may also refer to a number indicating the position of the image within the same series. The imaging start position may correspond to coordinates indicating the position where the examination (e.g., a type of imaging scan) begins. The image type may include information indicating whether the image is a slice image or a scout image in the examination. A file path refers to information indicating the location where medical images are stored in a specific database.

[0015] Medical imaging devices or other medical devices may use one or more medical data protocols for storing, transferring, and / or exchanging medical data. In particular, medical images may be converted into file formats compliant with the DICOM (Digital Imaging and Communications in Medicine) standard. The DICOM standard may include encrypting medical data into DICOM objects that contain images and other information such as demographic data, medical patient information, device data, examination details, basic image metadata, personally identifiable information (PII), and / or personal health information (PHI). Thus, DICOM objects may be used with databases to protect and retain sensitive PII and / or PHI data. In this way, the DICOM protocol can enable communication and data exchange between different systems such as user devices, medical imaging devices, medical image management devices, telemedicine servers, cloud servers, or processing servers for performing IP operations or AI processing operations. DICOM objects may also include video clips and cineloops.

[0016] One or more IP functions (e.g., IP function Y(173), IP function Z(151)) may be applied to medical images during IP operation. For example, an IP function may include various techniques and algorithms applied to digital images to analyze, enhance, and / or optimize image characteristics such as sharpness, contrast, size, and format. More specifically, an IP technique may treat a medical image as a data signal or a matrix and use it in a digital signal processing technique or a matrix operation technique, respectively. An IP function may be implemented using various parallel processing techniques, such as General Purpose Computing on Graphics Processing Units (GPGPU) or other multi-core techniques of a high-performance computing (HPC) server.

[0017] Furthermore, IP functionality may include one or more compression techniques. Image compression can include both lossy and lossless compression. Images reconstructed using lossy compression are visually similar to the original image, but the compressed image is not identical to the original. On the other hand, medical images reconstructed using lossless compression are identical to the original medical image. Because lossy compression can result in the loss of patient information, lossless compression is typically used for medical images. Examples of medical image compression include JPEG10 compression, JPEG-200011 compression, Huffman coding, arithmetic coding, Golon coding, run-length coding, LZW (Lempel-Ziv-Welch) coding, and predictive coding. Since medical images may focus on a region of interest, some image compression functions apply lossless compression to the region of interest and lossy compression to the region of non-interest. In particular, JPEG10 compression uses the discrete cosine transform, and JPEG-200011 compression uses the wavelet transform. Medical image compression also utilizes other transformation domains, such as discrete orthogonal Stockwell transforms and integer wavelet transforms.

[0018] Other IP functions are specific to various modalities. For example, computed tomography includes various reconstruction algorithms that combine multiple projection images to create a 3D image. Examples of CT reconstruction algorithms include iterative reconstruction (IR), filtered backprojection (i.e., backprojecting updated images onto themselves until the difference between images reaches a set value), weighted backprojection (WBP) (i.e., reconstructing a 3D volume from a 2D projection), and simultaneous iterative reconstruction (SIRT). Similarly, MRI devices can acquire raw MRI data that is not in image space. Thus, IP functions can perform image reconstruction processing to generate clinically interpretable MRI images. For example, MRI image reconstruction functions may include noise pre-whitening for phased array data acquisition, interpolation between square pixels, raw data filtering to reduce Gibbs ringing artifacts, Fourier transforms to connect raw MRI data to image space, and combinations of phased array coils.

[0019] IP operations also include various X-ray image processing steps. For example, in dynamic X-ray imaging, one or more X-ray images can be used to visualize and quantify the movement of organs. As such, dynamic X-ray imaging may identify various lung diseases such as chronic obstructive pulmonary disease (COPD), pneumonia, and lung cancer by providing new information regarding the respiratory movement of the diaphragm, ribs, and other structures in the lung region. Furthermore, dynamic X-ray imaging may use various IP techniques for functional imaging of ventilation, blood circulation, and other bodily functions. Unlike static imaging, dynamic X-ray imaging can obtain more information, thereby accelerating the elucidation of various pathological conditions, the initiation of treatment, and pre- and post-treatment evaluations, and is expected to contribute to improving medical efficiency and reducing medical expenses. More specifically, in dynamic X-ray imaging, such IP may be implemented using recognition processing for lungs and hearts, temporal filtering processing, reference frame subtraction processing, and the like.

[0020] Furthermore, a medical image management device (e.g., medical image management device Y(170)) may include hardware and / or software for deploying a medical system on a cloud network (e.g., medical data network X(100)). For example, a medical image management device is a server that displays medical images using a network of servers, applications, and storage devices hosted on the Internet. In particular, a medical image management device may control the presentation and transmission of various medical image files to various healthcare professionals via various user devices (e.g., user device A(110), user device B(120), user device C(130)). An example of a medical image management device is a PACS (Picture Archiving and Communication System) used in various medical institutions. Thus, a medical image management device may have hardware and / or software for storing, retrieving, managing, and distributing medical images to various user devices, medical devices, etc. In other words, medical image management systems can enable the transmission of medical image data and other medical information (e.g., clinical reports and personally identifiable information for immediate use) between physicians, patients, medical professionals, and external consultants. Thus, medical image management systems can securely transfer a patient's personal medical image information, as opposed to searching through physical files.

[0021] The medical image management apparatus can further provide a smooth image diagnosis system that checks whether medical images have been uploaded to the cloud and acquires real-time information on whether IP processing and artificial intelligence processing have been completed on the cloud service provider side. In particular, the medical image management apparatus may transmit, via a computer network, various status messages (e.g., status message B(123), status message C(132), status message O(183)) used via a graphical user interface on a user device (for example, status message B(123) is displayed on a web browser B(122) on a display device B(121)) or an application programmable interface. The status messages may indicate the status of various processing operations being performed on medical images within the cloud. Automating real-time status updates enables various medical systems and medical professionals to achieve smooth inter-group collaboration and remote access to medical data. For example, a server (e.g., a medical image management apparatus or a processing server) performs server-side rendering, in which case the server performs the rendering processing instead of the user device displaying the medical data. Since rendering is performed on a medical image (e.g., medical image Z(153)) at the server, DICOM data may not be transmitted to the user device that is actually displaying the specific medical image and related information. Furthermore, server-side rendering improves performance speed on the user device and minimizes hardware technical requirements on the user device, regardless of the file size from new imaging modalities.

[0022] Figure 4 shows an example of a medical image management device D(470) which includes a medical image database (431) connected to an image receiving unit (432) such as a communication interface for communicating with various network protocols. The medical image management device D(470) further includes an image upload unit (433), an image import unit (434), a status monitoring unit (435), and a status verification unit (436). These various units may be various hardware and / or software modules configured to perform various functions with respect to a processing server D(450) in a cloud network D(400).

[0023] A medical image management system may also collect various types of medical data. For example, a medical image management system may store supplementary information along with medical image data. Supplementary information may include one or more tables in the database, such as a patient table, examination table, series table, and image table. For example, a medical image management system may have a database with the following table structure: Patient Information - Examination Information - Series Information - Image Information. Status attributes of IP operations and / or AI processing operations may be maintained under the Examination Information table in the medical image management system. For parent tables (in the case of series tables, the examination table is the parent), the parent table is updated based on the information in the child tables (for example, if the status of all child tables is "completed", the parent record will also show "completed"). For example, status information for one or more medical images being processed is stored under the "Examination Information" table. Figures 5 and 6 show an example of a parent-child table relationship using this table structure.

[0024] Furthermore, the medical image management system may send data (e.g., medical image M(181)) and / or commands (e.g., command N(182)) to one or more processing servers (e.g., processing server Z(150)). For example, the processing servers may be used to provide one or more cloud-native medical image processing applications. The processing servers may import data in DICOM P10 format from the medical image management system and provide an application-programmable interface (API) for low-latency retrieval and dedicated storage. While the processing servers are performing import operations, the medical image management system may receive various status messages, such as whether the medical image has been sent to the processing server, whether a particular processing operation is in progress, whether the processing operation has been completed, whether the medical image has been validated, whether the processing operation has failed, or whether other errors have occurred.

[0025] Various queues (e.g., processing queue Y(176)) may be used to manage the uploading and / or processing of various medical images. For example, a processing queue may be used to manage the uploading of multiple medical image files to a network location within a cloud network. Similarly, a processing queue may determine the priority of different medical images under IP operations and / or AI processing operations. A processing queue may also follow various queue algorithms, such as first-in, first-out, to determine the order (sequence) of medical image processing.

[0026] Medical image management systems and / or user devices may coordinate processes performed using commands. Examples of commands include receiving network messages transmitted via inter-machine network protocols or control signals that automatically trigger one or more server operations. Commands respond to requests for the execution of specific processing operations or requests for status information regarding one or more scheduled processing operations. Similarly, commands may terminate or adjust processing operations in real time (for example, by changing image processing parameters or AI parameters).

[0027] A medical image management system may include hardware and / or software having the capability to perform various AI functions (e.g., AI function Y(174), AI function Z(152)). For example, the AI ​​function may be used for various AI processing operations, such as the implementation of inference operations as well as the generation and / or updating of machine learning models (e.g., machine learning model Y(171)). For example, a machine learning model may take one or more medical images as input, as well as other information (e.g., patient information, examination information, series information, etc.). These inputs can be used by the machine learning model to output predictive diagnoses and other medical information (e.g., boundaries of specific organs in X-rays, CT scans, MRI scans). Furthermore, various types of machine learning models may be trained, such as convolutional neural networks, deep neural networks, U-net models, recurrent neural networks, support vector machines, decision trees, inductive learning models, deductive learning models, supervised learning models, unsupervised learning models, and reinforcement learning models. For example, in a deep neural network, layers of neurons are trained with a predetermined list of features based on the output of previous network layers. Therefore, as the data passes through the deep neural network, more complex features of the data are identified by neurons in later layers. Furthermore, two or more different types of machine learning models are integrated into a single machine learning architecture. For example, machine learning models include decision trees and neural networks. In some embodiments, the medical image management device or processing server may generate augmented or synthetic data to produce large amounts of interpreted data for training a particular model.

[0028] With regard to support vector machines, a support vector machine may be a machine learning model trained using supervised machine learning algorithms. For example, a support vector machine may provide data analysis on various input features on which classification and regression analyses are performed. More specifically, a support vector machine may determine a hyperplane that separates a dataset into different classes and also determine various points (i.e., support vectors) that are closest to different classes. Furthermore, a support vector machine may use one or more kernel functions to transform the data into a desired format for further processing. The term "kernel" can also refer to a set of mathematical functions that provide a window for manipulating the input data. In other words, kernel functions can transform a training set of data so that a nonlinear decision surface can be transformed into a linear equation in a higher-dimensional space. Examples of kernel functions include Gaussian kernel functions, Gaussian kernel radial basis functions (RBFs), sigmoid kernel functions, polynomial kernel functions, and linear kernel functions.

[0029] In the case of artificial neural networks, for example, a neural network may contain one or more hidden layers, and each hidden layer contains one or more neurons. Neurons are modeling nodes or objects that loosely mimic the nerve cells of the human brain. Specifically, neurons can combine data inputs with a set of coefficients, or network weights, to adjust the data inputs. These network weights amplify or decrease the value of particular data inputs, thereby assigning importance to various data inputs for the task being modeled. Through machine learning, a neural network may determine which data inputs should be given higher priority when determining one or more specified outputs of the neural network. Similarly, these weighted data inputs are summed, and this sum is transmitted to other hidden layers in the neural network through the neuron's activation function. Thus, the activation function may determine whether, and to what extent, the neuron's output proceeds to other neurons whose outputs may be weighted again for use as input to the next hidden layer.

[0030] Turning to convolutional neural networks (CNNs), a type of artificial neural network used in computer vision and image recognition, for example, for processing pixel data. For instance, a convolutional neural network can include the ability to apply filters to its input (e.g., an input image) that result in specific activations. Repeated application of filters yields an output map of activations called a feature map. The feature map can show the location and intensity of one or more features detected in the input to the convolutional neural network. Thus, convolutional neural networks have the ability to automatically learn multiple filters in parallel that are specific to the training dataset, under the constraints of certain predictive modeling problems such as image classification.

[0031] In some embodiments, the server uses one or more ensemble learning methods in relation to various machine learning models. For example, an ensemble learning method may use multiple types of machine learning models to obtain better predictive performance than a single machine learning model. In some embodiments, for example, an ensemble architecture may combine multiple base models to produce a single machine learning model. One example of an ensemble learning method is a BAGGing model (i.e., BAGGing is a model that performs bootstrapping and aggregation operations). This combines predictions from multiple neural networks to add a bias that reduces the variance of a single trained neural network model. Another ensemble learning method is stacking, which may involve fitting many different model types to the same data and combining various predictions using different machine learning models. In some embodiments, two or more different types of machine learning models are integrated into a single machine learning architecture. For example, the machine learning models may include support vector machines and neural networks.

[0032] In some embodiments, the model may be trained using various types of machine learning algorithms, such as the backpropagation algorithm (e.g., machine learning algorithm Y(172)). In the backpropagation algorithm, for each hidden layer of the neural network, the gradient is calculated in reverse, from the layer closest to the output layer to the layer closest to the input layer. Thus, the gradient may be calculated using the transpose of the weights of each hidden layer based on an error function (also called a “loss function”). The error function may be based on various criteria, such as the mean squared error function or a similarity function, and the error function may be used as a feedback mechanism to adjust the weights of the machine learning model.

[0033] In some embodiments, machine learning models are trained using multiple epochs. For example, an epoch is an iteration of the model through part or all of the training dataset. Thus, a single machine learning epoch corresponds to a specific batch of training data, and the training data is divided into multiple batches for multiple epochs. In this way, a machine learning model may be iteratively trained using epochs until the model achieves a predetermined criterion, such as a predetermined level of predictive accuracy, or training over a certain number of machine learning epochs or iterations. Therefore, better training of the model may lead to better predictions from the trained model.

[0034] Turning to recurrent neural networks (RNNs), a recurrent neural network can repeatedly perform a specific task on multiple data elements of an input sequence, and the output of the recurrent neural network depends on past calculations. Thus, a recurrent neural network can operate in memory or hidden cell states that provide information for use in the current cell calculation with respect to the current data input. For example, a recurrent neural network may resemble a chain structure of RNN cells, and different types of recurrent neural networks have different types of repeated RNN cells. Similarly, the input sequence may be time-series data, and the state of the hidden cells may have different values ​​at different time steps during the prediction or training operation. For example, while deep neural networks use different parameters in each hidden layer, recurrent neural networks have common parameters in the RNN cells and run over multiple time steps. To train a recurrent neural network, a supervised learning algorithm such as a backpropagation algorithm may be used. In some embodiments, the backpropagation algorithm is the time-backpropagation (BPTT) algorithm. Similarly, the BPTT algorithm may determine gradients for updating various hidden layers and neurons within a recurrent neural network in a manner similar to how it is used to train various deep neural networks. In some embodiments, the recurrent neural network is trained using a reinforcement learning algorithm, such as a deep reinforcement learning algorithm. For more information on reinforcement learning algorithms, see the description below.

[0035] Embodiments using different types of RNNs are possible. For example, classical RNNs, long-short memory (LSTM) networks, gated recurrent units (GRUs), stacked LSTMs with multiple hidden LSTM layers (i.e., each LSTM layer contains multiple RNN cells), attention-based recurrent neural networks (i.e., machine learning models can focus attention on specific elements in the input sequence), bidirectional recurrent neural networks (e.g., machine learning models with independent hidden layers, such as forward and reverse layers, that can learn simultaneously in both time directions), multidimensional LSTM networks, graph recurrent neural networks, grid recurrent neural networks, and so on. With respect to LSTM networks, an LSTM cell may include various output lines that carry vectors of information, for example, from the output of one LSTM cell to the input of another LSTM cell. Thus, an LSTM cell may include multiple hidden layers and various point units that perform calculations such as vector addition.

[0036] Regarding region-based convolutional neural networks, a region-based convolutional neural network (R-CNN) may acquire an input image or other image data in its input layer. The R-CNN can then perform a selective search function to extract various regions of interest (ROIs), which may correspond to predetermined boundaries (e.g., specific rectangles) of objects within the input image. For example, an input petrological image may contain 1000 regions of interest (or region proposals) to be analyzed by the R-CNN. After determining the image data for different regions, each image data from each region may be passed through the neural network to determine various output features, such as whether a particular region proposal corresponds to a heart object or a non-heart object. For each region's output feature, a collection of support vector machines may act as a classifier used to determine what types of objects are contained within each region. Furthermore, the various regions used by the R-CNN are sometimes called "region proposals," which identify small areas of image data that may contain the objects being searched for in the input image data. A selective search function may be used as appropriate to reduce the number of region proposals in the R-CNN. Furthermore, the convolutional neural network and support vector machine may be trained separately based on the classification function within R-CNN.

[0037] Furthermore, various types of region-based convolutional neural networks are conceivable. For example, R-CNN can be Fast R-CNN, Faster R-CNN, Mask R-CNN, or You-only-look-once (YOLO) network. While a standard R-CNN can independently determine the features of the neural network for each region of interest, Fast R-CNN can use the neural network only once for the entire image. At the end of the convolutional neural network (CNN), ROI pooling is performed, extracting each region from the network's output tensor, reshaping the output features, and then classifying the reshaped output features to determine, for example, lithology parameters or cutting parameters. Similar to a standard R-CNN, Fast R-CNN may also use selective search functions or processes to generate various region proposals. In Faster R-CNN, ROI generation may be integrated into the convolutional neural network itself. In a YOLO network, an image can behave similarly to a fully convolutional neural network by passing it through an FCNN once and outputting a specific prediction for a grid containing bounding boxes and class probabilities for those bounding boxes. In a Mask R-CNN, the Mask R-CNN may include object instance segmentation. Object instance segmentation may involve determining the segmentation of the mask for each object instance, as well as detecting the object class (e.g., whether part of the medical image data is bone or another organ). Similarly, machine learning models that perform only semantic segmentation are conceivable, such as distinguishing biological objects within medical image data or detecting the presence of other image object types.

[0038] One example of AI functionality is a positioning function that identifies the displacement region or range of a joint in a medical image. The positioning function may also be an inference function that identifies the local region of the joint in the image based on a machine learning model (for example, using a deep neural network). The positioning function may further perform segmentation processing of the identified local region based on machine learning to identify the displacement region. In this way, the positioning function may measure the distance from the joint center of the divided region and determine the maximum displacement range.

[0039] Other examples of AI functions include, firstly, left / right determination, an inference function that performs preprocessing such as changing the image size or adjusting the gradation and density of captured medical images. The left / right determination function may estimate the probability that a medical image is of the left or right side of a joint based on machine learning. Alternatively, the left / right determination function may determine the side with the higher probability as the side of the joint from which the medical image was actually taken. These AI functions may require high-speed processing because the output results need to be displayed on the user device immediately after X-ray image acquisition.

[0040] Figure 1 shows various configurations of the components, but other configurations can be used without departing from the scope of this disclosure. For example, various components in Figure 1 can be combined to create a single component. As another example, a function performed by one component may be performed by two or more components.

[0041] Figure 2 illustrates a general method for providing status messages for various processing operations. The blocks in Figure 2 can be executed by various components (medical image management device Y(170) or processing server Z(150)) as shown in Figure 1. Although the various blocks in Figure 2 are presented and described sequentially, those skilled in the art will understand that some or all of the blocks may be executed in different orders, combined, omitted, and some or all of the blocks may be executed in parallel. Furthermore, the blocks may be executed actively or passively.

[0042] In block 200, one or more medical images generated by one or more medical imaging devices are received via the network. For example, the medical imaging devices may be X-ray machines, MRI machines, or other imaging devices connected to the user device. After acquiring medical images using one or more medical imaging devices, the medical images may be uploaded to a medical image management device or another server. Alternatively, the user device may acquire one or more medical images from the network, such as a local database, and upload the medical images accordingly.

[0043] In block 205, one or more medical images are inserted into a processing queue. For example, the processing queue may be an upload queue located on a user device, or a medical image management device that determines the order in which medical images are uploaded to a specific network location. Furthermore, the processing queue may specify the order in which different medical images undergo IP operations and / or AI processing operations. Thus, medical images may be inserted into the processing queue of a server that performs IP and / or AI processing.

[0044] In block 210, medical images are selected from the processing queue. For example, various medical images are iteratively selected by the medical image management device in a predetermined order in which various image files are uploaded to the processing server. Similarly, medical images are automatically selected by the medical image management device for further processing based on user selection in the graphical user interface or based on network requests.

[0045] In block 215, one or more IP operations and / or one or more artificial intelligence (AI) processing operations are performed on the selected medical image. For example, the IP operations and / or AI processing operations are selected by the user in connection with the upload of the selected image. Similarly, various processing operations are performed automatically in response to the image upload. For example, the medical image undergoes one or more types of image compression automatically before it can be used clinically on one or more user devices.

[0046] Figure 3 shows an example of communication between a hospital terminal (311) on a hospital network (310) and a medical image management device C (370) on a cloud network C (300). The hospital terminal (311) acquires images of different modalities, namely modality A (301) and modality B (302), and transmits them to the medical image management device C (370). The medical image management device then relays the medical images to a processing server C (350) for IP operations and / or AI processing operations. Examples of modalities include a type of treatment method, a type of medical device (e.g., a medical imaging device), and a type of strategy used to treat a patient's condition or injury. Modalities include physical, electrical, mechanical, and chemical modalities.

[0047] In block 220, status messages are sent for selected medical images based on the progress of one or more IP operations and / or one or more AI processing operations. In particular, medical images are identified within the graphical user interface as having the status attributes "QUEUE," "IN PROGRESS," "ERROR," "COMPLETE," "IP," and "ARTIFICIAL INTELLIGENCE PROCESSING." The "QUEUE" status may identify a medical image before uploading and / or before further progress. "IN PROGRESS" may identify a medical image during uploading, IP processing, and / or AI processing. The "ERROR" status corresponds to errors related to uploading, IP, and / or AI processing. Furthermore, some status messages specify the associated process, such as "IP ERROR" and "AI ERROR," which correspond to IP errors and AI processing errors, respectively. The status message may also specify whether a particular process has been completed, such as "IP Complete," "AI Complete," or "Upload Complete."

[0048] Furthermore, various status messages, such as notifications from a web browser or medical image management device, may be displayed on the user device. Figure 7 shows an example of a medical image management device E (770) communicating with a web server (775) using the Transmission Control Protocol (TCP) (785). Next, the web server (775) communicates with a user device (not shown) using Hypertext Transfer Protocol Secure (HTTPS) (780) and sends status messages to be displayed in a web browser (790). Thus, the user can communicate with the medical image management device through a web browser, and Figure 7 shows a display screen called the examination worklist, where the user can view examination-based information. In this list, the user can see the status of various medical images on the processing server.

[0049] Block 225 determines whether an error was detected during the execution of one or more IP operations and / or one or more AI processing operations. For example, the IP function and / or AI function may output an error message indicating an error during processing. Similarly, the image output from the IP or AI processing operation is verified by the medical image management device or processing server using one or more error detection functions to determine whether one or more errors occurred during processing. In the case of a compression operation, the error detection function may compare the compressed image with the original medical image to determine the presence of one or more errors. Similarly, test data may be used with a trained machine learning model to verify whether the model has achieved the desired accuracy.

[0050] In block 230, an error message regarding the selected image is sent. For example, the processing server may relay the error message to the medical image management device. The medical image management device may then send one or more status messages indicating errors to one or more user devices over the network.

[0051] In block 235, one or more commands relating to one or more IP operations and / or one or more AI processing operations are sent. For example, a user device may receive a user selection of one or more commands in a graphical user interface in response to receiving a status message indicating one or more errors. More specifically, the user may modify one or more image parameters of an IP operation and / or one or more AI parameters of an AI processing operation to resolve an error. For example, a compression process may be selected based on a specific modality for an inaccurate medical image. Upon reviewing the error message, a different type of compression has been selected for the modality used to obtain the uploaded medical image. In particular, the user device may send a request to a medical image management device or processing server to adjust the corresponding image parameters of the IP operation. Upon receiving this request, the adjusted IP operation is automatically restarted using the adjusted parameters.

[0052] In the case of AI processing operations, the user device may send different commands to change the type of inference operation and various AI parameters associated with each AI processing operation. For example, the user may select a different machine learning model in response to a specific error. If an error occurs during a training operation (e.g., the model does not achieve the desired level of predictive accuracy), the user may adjust the operation to use a different training dataset, a different initial model, and various hyperparameters associated with the trained model. In inference operations, the user may select a different inference operation to be performed on a selected medical image (for example, an inference operation to diagnose heart disease may produce errors if a medical image of a fractured femur is used). Thus, the user may analyze the selected medical image during AI processing using one or more error detection functions to decide whether to change the type of inference operation.

[0053] Block 240 determines whether one or more IP operations and / or one or more AI processing operations have completed. Depending on the execution of the IP operations and / or AI processing operations, the user device may display a status message indicating that each operation has been completed. Subsequently, one or more user devices may access output data associated with the specific processing operation. If the medical images are compressed, doctors and other medical professionals can access the compressed images for faster access. If an inference operation has been performed, the user can access the predictive data obtained from the inference operation. If a training operation has been performed, the trained model will be available for predicting patient data after completion.

[0054] In block 245, a status message is sent regarding the progress of the selected medical image. For example, the status message may simply indicate whether a particular IP operation or AI processing operation is still being performed. Alternatively, the status message may indicate the degree of completion of a particular operation (e.g., 70% complete).

[0055] Block 250 receives one or more commands relating to one or more IP operations and / or one or more AI processing operations. For example, the user may decide to change the type of IP operation and / or AI processing operation before completing it. Thus, the user device may send a command to terminate an ongoing processing operation, such as using a different selected image for the processing operation or changing the type of processing operation (e.g., adjusting the parameters of the IP operation).

[0056] In block 260, a status message is sent for the selected medical image based on the completion of one or more IP operations and / or one or more AI processing operations. When the medical image management device determines that an IP operation or AI processing operation has been completed, it may send a status message indicating completion of processing to the user device. In this way, the selected medical image may be made accessible to a wider range of users and may also be further processed (for example, after an IP operation is completed, one or more AI processing operations may be performed on the selected image to determine medical diagnostic information for a clinician). Figure 8A shows an example of a decision tree for a medical image that traverses various possible status messages based on queues and IP operations. Figure 8B shows an example of a decision tree based on traversing IP operations and AI processing operations.

[0057] Block 270 determines whether another medical image is in the processing queue. For example, a medical image management device or processing server may proceed with processing through a specific queue until all IP processing and / or AI processing operations have been performed. If another medical image is in the processing queue, blocks 210 to 260 may be repeated.

[0058] Figure 9A shows a computing system (900) which may include any combination of mobile, desktop, server, router, switch, embedded device, or other types of hardware. The computing system (900) may include one or more computer processors (902), non-persistent memory (904) (e.g., volatile memory such as random access memory (RAM) or cache memory), persistent memory (906) (e.g., hard disk, optical drive such as a compact disc (CD) drive or digital versatile disc (DVD) drive, flash memory, etc.), and communication interfaces (912) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.). The computer processor (902) may be an integrated circuit for processing instructions. For example, the computer processor may be one or more cores or microcores of a processor. The computing system (900) may also include one or more input devices (910), such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. The communication interface (912) may include an integrated circuit for connecting the computing system (900) to a network (for example, a local area network (LAN) or a wide area network (WAN) such as the Internet or a cellular network).

[0059] Furthermore, the computing system (900) may include one or more output devices (908), such as a screen (e.g., a liquid crystal display (LCD), a plasma display, a touchscreen, a projector, or other display device), a printer, external storage, or any other output device. One or more output devices may be the same as or different from the input devices. The computing system (900) may implement and / or connect to a data repository. For example, a type of data repository is a database. A database is a collection of information configured to facilitate retrieval, modification, reorganization, and deletion of data. A database management system (DBMS) is a software application that provides an interface for users to define, create, query, update, and manage databases.

[0060] Software instructions in the form of computer-readable program code for performing various functions may be stored, in whole or in part, temporarily or permanently on non-transient computer-readable media such as memory devices, diskettes, tapes, flash memory, physical memory, or other computer-readable storage media. Specifically, the software instructions may correspond to computer-readable program code configured to perform various functions when executed by a processor(s).

[0061] The computing system (900) in Figure 9A may be connected to or part of a network. For example, as shown in Figure 9B, the network (920) may include multiple nodes (e.g., node X (922), node Y (924)). Each node may correspond to a computing system, such as the computing system shown in Figure 9A. Alternatively, a combination of nodes may correspond to the computing system shown in Figure 9A. The nodes in the network (920) (e.g., node X (922), node Y (924)) may be configured to provide services to a client device (926). The nodes may include the ability to receive requests from the client device (926) and send responses to the client device (926). The client device (926) may be a computing system, such as the computing system shown in Figure 9A.

[0062] The computing system (900) may be implemented as part of a cloud computing system. For example, a node may be part of a cloud computing system. For example, a cloud computing system may include remote servers along with various other cloud components such as cloud storage units and edge servers. In this way, a cloud computing system can have different functions distributed across multiple locations from a central server, running using an internet connection.

[0063] The computing system may further include the ability to receive data from the user. For example, the user may send data via a graphical user interface (GUI) on the user device. The user may send data via the graphical user interface by selecting one or more graphical user interface components or inserting text or other data into graphical user interface widgets using a touchpad, keyboard, mouse, or other input device. Depending on the selection of a particular item, information about that item may be retrieved by the computer processor from persistent or non-persistent storage. Depending on the item selected by the user, the content of the retrieved data about that item is displayed on the user device.

[0064] Although only a few exemplary embodiments have been described in detail above, those skilled in the art will readily understand that many modifications are possible in the exemplary embodiments without substantially departing from the present invention. Accordingly, all such modifications are intended to fall within the scope of this disclosure as defined in the following claims.

Claims

1. The server receives a first medical image generated using a medical imaging device from a user device via the network, Performing a first image processing operation on the first medical image based on at least one image processing parameter, The server determines whether the first image processing operation on the first medical image has been completed, A method comprising: the server sending a first status message to the user device via the network in response to determining that the first image processing operation on the first medical image has been completed.

2. Displaying the first status message on the user device's display device, The user device, in response to displaying the first status message to the server via the network, sends a request to the server, wherein the request adjusts one or more image processing parameters in the first image processing operation and generates the adjusted image processing operation. The method according to claim 1, further comprising performing the adjusted image processing operation.

3. Performing artificial intelligence (AI) processing operations on the second medical image, The server determines whether the AI ​​processing operation has been completed for the second medical image, The method according to claim 1, further comprising: sending a second status message in response to the server determining that the AI ​​processing operation has been completed for the second medical image.

4. The aforementioned AI processing operation is an inference operation, The second medical image is the input to the machine learning model. The method according to claim 3, wherein predictive medical data is generated as the output of the machine learning model by the AI ​​processing operation.

5. The aforementioned AI processing operation is a training operation for training a machine learning model, The second medical image mentioned above is part of the training dataset. The method according to claim 3, wherein the machine learning model is trained using the training dataset and the supervised learning algorithm.

6. The aforementioned server is a medical image management device, The first image processing operation is performed by a processing server different from the medical image management device. The first status message corresponds to the status information stored in the database of the medical image management device, The aforementioned database includes a patient table, a test table, a series table, and an image table. The method according to claim 1, wherein the status information is placed in a data field in the inspection table.

7. The aforementioned user device is a hospital terminal, The hospital terminal accesses the medical image management device to obtain supplementary information regarding the first medical image. The method according to claim 1, wherein the supplementary information includes status information relating to the image processing operation in the medical image management device.

8. The method according to claim 1, wherein the first image processing operation is performed using an image compression function.

9. The method according to claim 1, wherein the medical imaging device is an X-ray imaging device.

10. The server determines, based on the second medical image and error detection function, whether or not an error occurred in the second image processing operation. The method according to claim 1, further comprising the server sending a second status message to the user device via the network, in response to determining that an error has occurred in the second image processing operation, the error being identified.

11. The process of selecting a second image from among multiple images placed in a processing queue, wherein the second image is automatically selected based on the priority of the second images in the processing queue. The method according to claim 1, further comprising performing a second image processing operation on the second image.

12. The method according to claim 1, wherein the first status message is displayed in a web browser on the user device.

13. Multiple user devices, The first processing server, A system comprising a medical image management device connected to the plurality of user devices and the first processing server via a network, The aforementioned medical image management device includes a computer processor, The aforementioned medical image management device, Receiving a first medical image generated using a medical imaging device from one of the multiple user devices via the network, To determine whether the first image processing operation on the first medical image has been completed, A system configured to perform a method comprising: sending a first status message to the user device via the network in response to determining that the first image processing operation on the first medical image has been completed.

14. The user device includes a display device configured to display the first status message using a graphical user interface, The method further comprises adjusting one or more image processing parameters in the first image processing operation and generating the adjusted image processing operation, The system according to claim 13, wherein the adjusted image processing operation is performed by the first processing server.

15. The system further comprises a second processing server connected to the medical image management device via the aforementioned network and configured to perform artificial intelligence (AI) processing operations on the second medical image, The aforementioned medical image management device, Determine whether the AI ​​processing operation on the second medical image has been completed. The system according to claim 13, further configured to send a second status message in response to determining that the AI ​​processing operation has been completed for the second medical image.

16. The aforementioned AI processing operation is an inference operation, The second medical image is the input to the machine learning model. The system according to claim 15, wherein predictive medical data is generated as the output of the machine learning model by the AI ​​processing operation.

17. The first status message corresponds to the status information stored in the database of the medical image management device, The aforementioned database includes a patient table, a test table, a series table, and an image table. The system according to claim 13, wherein the status information is located in a data field within the inspection table.

18. The system according to claim 13, wherein the first status message is displayed in a web browser on the user device.

19. The system according to claim 13, wherein the medical imaging device is an X-ray imaging device.

20. A non-temporary computer-readable medium, When executed by a computer, Receiving medical images generated using a medical imaging device from a user device via a network, To determine whether the image processing operation performed by the processing server has been completed for the medical image, A status message is sent to the user device via the aforementioned network in response to the determination that the image processing operation on the medical image has been completed. A non-temporary computer-readable medium comprising instructions configured to perform a method comprising [a certain action].

Images