Sequential transmission of compressed medical image data

Abstract

A medical system (100, 300, 400) is disclosed that includes a local memory (118) storing local machine executable instructions (130) and a local computing system (110). Execution of the machine executable instructions also causes the computing system to receive (200) a feature vector (132, 134, 136) describing medical image data (410), where the feature vector is configured to be input into a decoder neural network (142), where the decoder neural network (142) is configured to receive a feature vector (132, 134, 136). The decoder neural network is configured to output an approximation (146) of medical image data when receiving as input at least a portion of a feature vector, where the feature vector includes an ordering (702) that assigns importance to elements of the feature vector; and sequentially transmitting (202) the constituent parts (132, 134, 136) of the feature vector to a remote computing system (120) via a network connection (114), where first the constituent parts of the feature vector having a higher importance are transmitted.

Description

Technical Field

[0001] The present invention relates to medical imaging, and more particularly to the transmission of medical image data. Background Art

[0002] Various tomographic medical imaging techniques such as magnetic resonance imaging (MRI), computed tomography, positron emission tomography, and single photon emission tomography enable detailed visualization of the anatomical structure of an object. A common feature of these imaging modalities is that the amount of measurement data used to reconstruct medical images can be quite large. Another common feature is that storing a large number of reconstructed medical images may require a large storage capacity.

[0003] International Patent Publication WO2021220008A1 discloses a computer-implemented method for lossy image or video compression, transmission, and decoding, the method comprising the steps of: (i) receiving an input image at a first computer system; (ii) using the first computer system to encode the input image using a first trained neural network to produce a latent representation; (iii) quantizing the latent representation using the first computer system to produce a quantized latent; (iv) entropy encoding the quantized latent into a bitstream using the first computer system; (v) transmitting the bitstream to a second computer system; (vi) the second computer system entropy decoding the bitstream to produce a quantized latent; (vii) the second computer system using a second trained neural network to produce an output image from the quantized latent, wherein the output image is an approximation of the input image. Summary of the Invention

[0004] The present invention provides a medical system, a computer program, and a method. Embodiments are given in the dependent claims.

[0005] As described above, when performing remote reconstruction of medical images or when retrieving reconstructed medical images or records from a storage location or data repository, a large amount of data needs to be transmitted. Embodiments may provide a more robust means of transmitting medical image data to a remote location. Embodiments may achieve this by receiving a feature vector describing the medical image data and then first transmitting the components of the feature vector that have a higher importance.

[0006] In one aspect, the present invention provides a medical system that includes a local memory storing local machine-executable instructions and a local computing system. The running of the machine-executable instructions causes the computing system to receive a feature vector that describes medical image data. The feature vector can be described as encoding the medical image data or providing a compression of the medical image data. The feature vector is configured to be input into a decoder neural network. The decoder neural network is configured to output an approximation of the medical image data when receiving at least a portion of the feature vector as input.

[0007] The feature vector includes an ordering that assigns importance to the elements of the feature vector. The ordering can be assigned in different ways. For example, various elements or members of the feature vector can include values that provide the ordering. In other cases, the feature vector can be binned into different bins or storage modules or locations to store the discretely assigned ordering. The running of the machine-executable instructions also causes the computing system to sequentially transmit components of the feature vector over a network connection to a remote computing system. First, the components of the feature vector having a higher importance as assigned in the ordering are transmitted. This embodiment can be beneficial because it can provide a means of transmitting components of the feature vector in a means that enables the reconstruction of an approximation of the medical image data.

[0008] In one example, the decoder neural network is the decoder component of an autoencoder neural network. Thus, there can be a corresponding encoder neural network that is the encoder component of the autoencoder neural network. The feature vector can be the latent space of the autoencoder. As used herein, the term decoder neural network can be replaced with the decoder component of the autoencoder. Also as used herein, the term encoder neural network can be replaced with the encoder component of the autoencoder.

[0009] Different types of autoencoders can be used to provide the encoder neural network and the decoder neural network. The encoder neural network and the decoder neural network can each be formed, for example, by a plurality of hidden layers that are a combination of convolutional layers, max pooling layers, batch normalization layers, ReLU layers, and / or fully connected layers. The autoencoder can also be a variational autoencoder or a vector quantization autoencoder. However, skip connections across the bottleneck of the autoencoder (e.g., used by variants of the Densenet implementation) are not used herein because they effectively "shorten" the required feature abstraction (and the associated size reduction).

[0010] For example, an encoder neural network can include an input layer configured to receive medical image data. The encoder neural network can then have a plurality of hidden layers formed by a combination of, for example, convolutional layers and / or fully connected layers. The output of the last hidden layer is then connected to an output layer configured to output a feature vector. The size of the feature vector is a choice that represents the degree of compression. In one case, the size of the feature vector can be chosen such that the number of its elements is 30% (between 25% and 35%) of the number of elements in the medical image data. Since a neural network can be trained to reproduce a particular type of data, accurate compression can be much higher than that of algorithms such as "zip".

[0011] Another approach is to have the feature vector have a number of elements that is closer in number to the number of elements in the original medical image data. If dropout is used to train the decoder neural network, the approximation of the medical image data provided when decompressing the feature vector can be very robust with respect to missing or incomplete data.

[0012] Similar to the example encoder neural network, a decoder neural network can include an input layer configured to receive the feature vector produced by the encoder neural network. The decoder neural network can then include a plurality of hidden layers formed by, for example, convolutional and / or fully connected layers. The last in the sequence of fully connected layers can be connected to an output layer of the decoder neural network, which is configured to output an approximation of the medical image data.

[0013] The decoder neural network and the encoder neural network can be trained together. For example, the encoder neural network and the decoder neural network essentially form an autoencoder, where the feature vector is equivalent to a latent space vector. A pair of encoder neural network and decoder neural network can be trained by obtaining examples of medical image data and inputting them into the encoder neural network and receiving the feature vector. The feature vector is then input into the decoder neural network, and then its output is compared with the original medical image data input into it.

[0014] To enable the decoder neural network to output an approximation when only a component of the feature vector is input, during training, components of the feature vector can be dropped out. For example, they can be set to zero, which is a non-value. In some cases, during training, an ordering can be assigned to the feature vector during training. The elements to be selected for dropout can be, for example, those with a lower ordering. This can provide a pair of encoder neural network and decoder neural network that have the ability to accurately approximate medical image data using only components of the feature vector.

[0015] When components of the feature vector are transmitted to a remote computing system, metadata describing the medical image data may also optionally be transmitted. If the medical image data is measurement data from a medical imaging system or a tomographic medical imaging system, the metadata may include a description or code describing how the medical image data was reconstructed into a medical image.

[0016] In another embodiment, the local memory further includes an encoder neural network. The running of the local machine-executable instructions also causes the local computing system to receive medical image data. The running of the local machine-executable instructions also causes the local computing system to receive a feature vector in response to inputting the medical image data into the encoder neural network. The running of the local machine-executable instructions also causes the local computing system to determine an ordering and assign the ordering to elements of the feature vector.

[0017] In another embodiment, the determination of the ordering includes: determining the magnitude of the elements of the feature vector and then assigning an ordering to the elements of the feature vector such that the ordering increases as the magnitude increases. The ordering can be, for example, a numerical value assigned to each of the elements of the feature vector. In other cases, the ordering can be a discreet assignment that can be a value, or it can also be determined by a location in a memory or storage device. For example, elements of the feature vector that are assigned a particular ordering can be grouped or stored together. This embodiment can provide an efficient means of assigning an ordering to the elements of the feature vector.

[0018] In another embodiment, the local memory further includes a saliency map neural network configured to receive the medical image data as input and provide, in response, an assignment as output, the assignment being an assignment of an ordering to elements of the feature vector. The running of the local machine-executable instructions further includes: receiving, from the saliency map neural network, an ordering of the elements of the feature vector in response to inputting the medical image data into the saliency map neural network. For example, this embodiment may be beneficial because it can provide a means of assigning importance to the various elements of the feature vector that does not depend solely on the size of the elements. This can provide a more accurate assignment.

[0019] The saliency map neural network can be formed, for example, by a number of hidden layers, such as convolutional layers, batch normalization layers, ReLU layers, and / or fully connected layers. Since this saliency map network is a classification / regression network independent of the decoder part of the autoencoder (as its purpose is to assign an ordering to the elements of the feature vector), it can also include "skip connections", so an architecture similar to Densenet can be used for this purpose.

[0020] However, it is worth noting that the training of the importance map requires data annotation indicating image regions with diagnostic relevance. These can be generated, for example, by manual annotation or by extracting relevant regions from radiologists' reports with the aid of natural language processing. In any case, this data may not be very abundant, making network architectures that require less training data (e.g., an EfficientNet-like implementation) potentially advantageous.

[0021] The training of the importance map network can include several reconstructions of the image from the feature vector using the decoder part of an autoencoder, where some noise is added to the individual components in an amount corresponding to the inverse (or squared inverse) of the importance map output. Then, the loss function will include the average of the differences between the reconstructed image and the original image, where using some metric (e.g., the L2 norm), the annotated regions are weighted higher compared to other regions.

[0022] In another embodiment, the importance map neural network is implemented as a separate neural network. As an example, the importance map neural network includes an input layer configured to receive the original medical image data. Then there are multiple hidden layers, e.g., including convolutional and / or fully connected layers. Then, the last one in the sequence of hidden layers is connected to an output layer configured to assign a ranking to each element of the feature vector.

[0023] In another embodiment, the importance map neural network is implemented as a constituent part of an encoder neural network. For example, the importance map neural network can be several convolutional layers that are separate from or share the input layer with the encoder neural network. As an example, the importance map neural network includes a connection between one of the hidden layers of the encoder neural network and an additional hidden layer belonging to the importance neural network. The additional hidden layer can be, for example, a convolutional and / or fully connected layer. Then, the last one in the sequence of hidden layers is connected to an output layer configured to assign a ranking to each element of the feature vector.

[0024] In another embodiment, the ranking includes assigning a ranking to an array of values of the elements of the feature vector. In this case, the ranking will be an assignment of which elements will be transmitted first.

[0025] In another embodiment, sorting includes assigning elements of a feature vector to a discrete number of storage locations. The discrete number of storage locations have different data retrieval latencies, and a lower retrieval latency is equivalent to a higher sort order. This embodiment can be particularly beneficial as it can provide an effective means of using memory locations with different latencies. For example, the most important data has a lower data latency and is thus available earlier for transmission. In some cases, it can also provide an economic benefit as data can be stored in data repositories or other storage locations on different media types or storage media. Thus, storage devices with more readily available data can be used for more important data and can, for example, have a higher cost. Less important data can be stored at locations or storage facilities with a greater latency time. This can, for example, provide a means of not only providing important portions of an image for faster reconstruction but also providing reduced storage costs.

[0026] In another embodiment, a feature vector is received by retrieving components of the feature vector from a discrete number of storage locations according to a sort order. In this case, components of the feature vector with a higher sort order are stored in locations with a lower latency and are retrieved first. As described above, this can provide an economy of various speeds and costs.

[0027] In another embodiment, the medical system further includes a remote memory that stores machine-executable instructions and a decoder neural network. As previously described, the encoder neural network and the decoder neural network can essentially form an autoencoder neural network. The medical system further includes a remote computing system. The running of the remote machine-executable instructions causes the remote computing system to sequentially receive components of the feature vector via a network connection. The running of the remote machine-executable instructions also causes the remote computing system to assemble the received components of the feature vector into a portion of the feature vector.

[0028] The running of the remote machine-executable instructions also causes the remote computing system to receive an approximation of medical image data in response to inputting the portion of the feature vector into the decoder neural network. The received components of the feature vector are input into the decoder neural network before all components of the feature vector have been received. This can, for example, provide a means of providing an initial approximation of the medical image data. In some cases, this approximation of the medical image data can, for example, enable an algorithm to begin a reconstruction process before all of the medical image data is available. In other cases, this also enables the reconstruction of medical image data if the transmission of the feature vector is interrupted.

[0029] In another embodiment, the medical image data is measurement data. The running of the remote machine-executable instructions also causes the remote computing system to reconstruct a preliminary medical image from the medical image data before all components of the feature vector have been received.

[0030] In different examples, the medical image data can take different forms. In one instance, it can be measurement data from a magnetic resonance imaging system. In this case, the measurement data can be k-space data. In other examples, such as for parallel imaging magnetic resonance imaging protocols, there are individual coil images generated for each coil element. In this case, the preliminary medical image can be a collection of individual coil images or a plurality of individual coil images from a parallel imaging magnetic resonance imaging profile.

[0031] In still other cases, the preliminary medical image can be data from a computed tomography (CT) or cone beam computed tomography (CBCT) system. For example, they can be CT profiles or CBCT images.

[0032] In another embodiment, the medical image data is a medical image. The running of the machine-executable instructions also causes the computing system to present an approximation of the medical image. For example, this can be particularly useful in cases where the feature vectors are sorted by distributing them into different bins or different storage locations with different delays. For example, this can enable the invocation of data and provide an initial reconstruction that can be used to evaluate whether it is desirable to retrieve all of the data.

[0033] In another embodiment, before all components of the feature vector have been received at the remote computing system, the assembly of the received components of the feature vector, the approximation of the medical image data, and the presentation of the approximation of the medical image data are repeated multiple times. The running of the remote machine-executable instructions also causes the remote computing system to receive a stop transmission instruction in response to the rendering of the approximation of the medical image data, and to transmit the stop transmission instruction to the local computing system via a network connection.

[0034] The running of the local machine-executable instructions also causes the local computing system to receive the stop transmission instruction via the network connection, and then to stop transmitting the components of the feature vector in response to receiving the stop transmission instruction. This embodiment can be particularly beneficial in terms of the amount of data transferred between the local computing system and the remote computing system when it is not necessary to transmit the entire feature vector.

[0035] In another aspect, the present invention provides a medical system that includes a remote memory storing remotely machine-executable instructions and a decoder neural network. The decoder neural network is configured to output an approximation of medical image data in response to receiving at least a portion of a feature vector as input. The medical system also includes a remote computing system. The running of the remotely machine-executable instructions causes the remote computing system to sequentially receive constituent parts of the feature vector via a network connection. The running of the remotely machine-executable instructions also causes the remote computing system to assemble the received constituent parts of the feature vector into a portion of the feature vector. The running of the remotely machine-executable instructions also causes the remote computing system to receive an approximation of medical image data in response to inputting the portion of the feature vector into the decoder neural network. The portion of the feature vector is input into the decoder neural network before all constituent parts of the feature vector have been received.

[0036] In another aspect, the present invention provides a computer program that includes locally machine-executable instructions for running by a local computing system. The running of the machine-executable instructions causes the computing system to receive a feature vector describing medical image data. The feature vector is configured to be input into a decoder neural network. The decoder neural network is configured to output an approximation of medical image data when receiving at least a portion of the feature vector as input. The feature vector includes an ordering that assigns importance to elements of the feature vector. The running of the machine-executable instructions also causes the local computing system to sequentially transmit constituent parts of the feature vector to a remote computing system via a network connection. The constituent parts of the feature vector having higher importance are transmitted first.

[0037] In another aspect, the present invention provides a method. The method includes receiving, via a local computing system, a feature vector describing medical image data. The feature vector is configured to be input into a decoder neural network. The decoder neural network is configured to output an approximation of medical image data when receiving at least a portion of the feature vector as input. The feature vector includes an ordering that assigns importance to elements of the feature vector. The method also includes sequentially transmitting, by the local computing system via a network connection, constituent parts of the feature vector to a remote computing system. The constituent parts of the feature vector having higher importance are transmitted first.

[0038] The method also includes sequentially receiving, by the remote computing system via a network connection, constituent parts of the feature vector. The method also includes assembling, by the remote computing system, the received constituent parts of the feature vector into the portion of the feature vector. The method also includes receiving an approximation of medical image data in response to inputting, by the remote computing system, the portion of the feature vector into the decoder neural network. The received constituent parts of the feature vector are input into the decoder neural network before all constituent parts of the feature vector have been received.

[0039] It should be understood that one or more of the foregoing embodiments of the present invention can be combined, provided that the combined embodiments are not mutually exclusive.

[0040] As those skilled in the art will recognize, aspects of the present invention can be implemented as an apparatus, a method, or a computer program product. Accordingly, aspects of the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects (collectively referred to herein as "circuitry", "module", or "system"). In addition, aspects of the present invention can take the form of a computer program product embodied in one or more computer-readable media having computer-executable code embodied thereon.

[0041] Any combination of one or more computer-readable media can be utilized. The computer-readable media can be a computer-readable signal medium or a computer-readable storage medium. As used herein, "computer-readable storage medium" encompasses any tangible storage medium that can store instructions executable by a processor of a computing device or a computing system. A computer-readable storage medium can be referred to as a computer-readable non-transitory storage medium. A computer-readable storage medium can also be referred to as a tangible computer-readable medium. In some embodiments, the computer-readable storage medium can also be capable of storing data accessible by a computing system of a computing device. Examples of computer-readable storage media include, but are not limited to: floppy disks, magnetic hard disk drives, solid state drives, flash memory, USB thumb drives, random access memory (RAM), read-only memory (ROM), optical disks, magneto-optical disks, and register files of a computing system. Examples of optical disks include compact disks (CDs) and digital versatile disks (DVDs), such as, CD-ROM, CD-RW, CD-R, DVD-ROM, DVD-RW, or DVD-R disks. The term computer-readable storage medium also refers to various types of recording media that can be accessed by a computer device via a network or communication link. For example, data can be retrieved over a modem, the Internet, or a local area network. Any suitable medium can be used to transmit the computer-executable code embodied on the computer-readable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

[0042] A computer-readable signal medium can include a propagated data signal having computer-executable code embodied therein, for example, in a baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0043] "Computer memory" or "memory" is an example of a computer-readable storage medium. Computer memory is any memory that can be directly accessed by a computing system. "Computer storage device" or "storage device" is another example of a computer-readable storage medium. A computer storage device is any non-volatile computer-readable storage medium. In some embodiments, a computer storage device can also be a computer memory, or vice versa.

[0044] As used herein, a "computing system" encompasses electronic components capable of running a program or machine-executable instructions or computer-executable code. References to a computing system that include examples of a "computing system" should be construed as possibly including more than one computing system or processing core. The computing system can be, for example, a multi-core processor. A computing system can also refer to a collection of computing systems within a single computer system or distributed across multiple computer systems. The term computing system should also be construed as possibly referring to a collection or network of computing devices each including a processor or computing system. Machine-executable code or instructions can be run by multiple computing systems or processors that can be within the same computing device or can even be distributed across multiple computing devices.

[0045] Machine-executable instructions or computer-executable code can include instructions or programs that cause a processor or other computing system to perform aspects of the present invention. The computer-executable code for performing operations for aspects of the present invention can be written in any combination of one or more programming languages and compiled into machine-executable instructions, the one or more programming languages including object-oriented programming languages such as Java, Smalltalk, C++, etc. and conventional procedural programming languages such as the "C" programming language or similar programming languages. In some instances, the computer-executable code can take the form of a high-level language or take a pre-compiled form and be used in conjunction with an interpreter that generates machine-executable instructions at runtime. In other instances, machine-executable instructions or computer-executable code can take the form of programming for a programmable logic gate array.

[0046] The computer-executable code can run entirely on the user's computer, partially on the user's computer (as a stand-alone software package), partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0047] Aspects of the present invention are described with reference to the flowcharts, illustrations, and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that, when applicable, each block or component of the blocks of the flowcharts, illustrations, and / or block diagrams can be implemented by computer program instructions in the form of computer-executable code. It should also be understood that, when not mutually exclusive, combinations of blocks from different flowcharts, illustrations, and / or block diagrams can be combined. These computer program instructions can be provided to a computing system of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus that produces a machine, such that the instructions run via the computing system of the computer or other programmable data processing apparatus create a unit for implementing the functions / actions specified in the flowchart and / or one or more block diagram blocks.

[0048] These machine-executable instructions or computer program instructions can also be stored in a computer-readable medium, which can direct a computer, other programmable data processing apparatus, or other device to work in a specific manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in the flowchart and / or one or more block diagram blocks.

[0049] The machine-executable instructions or computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable device, or other device, thereby generating a computer-implemented process, such that the instructions running on the computer or other programmable device provide a process for the functions / actions specified in the flowchart and / or one or more block diagram blocks.

[0050] As used herein, "user interface" is an interface that allows a user or operator to interact with a computer or computer system. A "user interface" may also be referred to as a "human-machine interface device". The user interface can provide information or data to the operator and / or receive information or data from the operator. The user interface can enable input from the operator to be received by the computer and can provide output from the computer to the user. In other words, the user interface can allow the operator to control or manipulate the computer, and the interface can allow the computer to indicate the effects of the operator's control or manipulation. Displaying data or information on a display or graphical user interface is an example of providing information to the operator. Receiving data via a keyboard, mouse, trackball, touchpad, pointing stick, graphics tablet, joystick, gamepad, webcam, headset, pedal, wired glove, remote control, and accelerometer are all examples of user interface components that implement the receiving of information or data from the operator.

[0051] As used herein, "hardware interface" encompasses interfaces that enable a computing system of a computer system to interact with and / or control external computing devices and / or apparatuses. The hardware interface can allow the computing system to send control signals or instructions to the external computing devices and / or apparatuses. The hardware interface can also enable the computing system to exchange data with the external computing devices and / or apparatuses. Examples of hardware interfaces include, but are not limited to: Universal Serial Bus, IEEE 1394 port, parallel port, IEEE 1284 port, serial port, RS-232 port, IEEE-488 port, Bluetooth connection, wireless local area network connection, TCP / IP connection, Ethernet connection, control voltage interface, MIDI interface, analog input interface, and digital input interface.

[0052] As used herein, "display" or "display device" encompasses output devices or user interfaces suitable for displaying images or data. The display can output visual, audio, and / or tactile data. Examples of displays include, but are not limited to: computer monitors, television screens, touchscreens, tactile electronic displays, braille screens,

[0053] cathode ray tubes (CRTs), storage tubes, bistable displays, electronic paper, vector displays, flat panel displays, vacuum fluorescent displays (VFs), light emitting diode (LED) displays, electroluminescent displays (ELDs), plasma display panels (PDPs), liquid crystal displays (LCDs), organic light emitting diode displays (OLEDs), projectors, and head-mounted displays.

[0054] Measurement data is defined herein as measurement data of an object recorded by a tomographic medical imaging system. The medical imaging data can be reconstructed into medical images. A medical image is defined herein as a reconstructed two - or three - dimensional visualization of anatomical data contained within the medical imaging data. This visualization can be performed using a computer.

[0055] Depending on the context, the term medical image data can refer to measurement data or a medical image. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Preferred embodiments of the present invention will be described hereinafter only by way of example and with reference to the accompanying drawings, in which:

[0057] Figure 1 An example of a medical system is illustrated;

[0058] Figure 2 Shows a flowchart of a method of a medical system illustrating the use of Figure 1 or Figure 3 ;

[0059] Figure 3 Another example of a medical system is illustrated;

[0060] Figure 4 Another example of a medical system is illustrated;

[0061] Figure 5 Shows a flowchart of a method of a medical system illustrating the use of Figure 4 ;

[0062] Figure 6 An example of an encoder neural network and a decoder neural network is illustrated;

[0063] Figure 7 Another example of an encoder neural network and a decoder neural network is illustrated; and

[0064] Figure 8 Another example of an encoder neural network and a decoder neural network is illustrated;

[0065] LIST OF REFERENCE NUMERALS

[0066] 100 Medical system

[0067] 102 Transmitter component

[0068] 104 Receiver component

[0069] 106 Local computer

[0070] 108 Remote computer

[0071] 110 Local computing system

[0072] 112 Local network interface

[0073] 114 Network connection

[0074] 116 Local hardware interface

[0075] 118 Local memory

[0076] 120 Remote computing system

[0077] 122 Remote network interface

[0078] 124 User interface

[0079] 126 Remote memory

[0080] 130 Local machine-executable instructions

[0081] 132 First component of the feature vector

[0082] 134 Second component of the feature vector

[0083] 136 Third component of the feature vector

[0084] 140 Remote machine-executable instructions

[0085] 142 Decoder neural network

[0086] 144 Incomplete feature vector

[0087] 146 Approximation of medical image data

[0088] 200 Receive a feature vector describing medical image data, wherein the feature vector is configured to be input into a decoder neural network

[0089] 202 Sequentially transmit the components of the feature vector to a remote computing system via a network connection

[0090] 204 Sequentially receive the components of the feature vector via the network connection

[0091] 206 Combine the received components of the feature vector into the part of the feature vector

[0092] 208 Receive an approximation of the medical image data in response to inputting the part of the feature vector into the decoder neural network

[0093] 300 Medical system

[0094] 302 First storage location

[0095] 304 Second storage location

[0096] 306 Third storage location

[0097] 400 Medical system

[0098] 402 Medical imaging system

[0099] 404 Object

[0100] 406 Object support

[0101] 408 Imaging area

[0102] 410 Medical image data

[0103] 412 Encoder neural network

[0104] 414 Feature vector

[0105] 416 Preliminary medical image

[0106] 500 Receive the medical image data.

[0107] 502 Receive the feature vector in response to inputting the medical image data into the encoder neural network.

[0108] 504 Determine the sorting and assign the sorting to the elements of the feature vector.

[0109] 600 Functional autoencoder

[0110] 700 Importance map neural network

[0111] 702 Sorting

[0112] 800 Importance map neural network layer Detailed implementation

[0113] Elements with similar numbers in these figures are either equivalent elements or perform the same function. If the functions are equivalent, the previously discussed elements will not be discussed in the subsequent figures unnecessarily.

[0114] Figure 1Illustrates an example of a medical system 100. The medical system 100 includes a transmitter component 102 and a receiver component 104. The transmitter component 102 in this example is shown to include a local computer 106. The local computer 106 is intended to represent one or more computers. The local computer 106 is shown to include a local computing system 110. The local computing system 110 is intended to represent one or more computing or computer cores. The local computing system 110 is shown to be connected to a local network interface 112, which is used to form a network connection 114 with a remote computer 108. The local computer 106 is also shown to include an optional local hardware interface 116. The local hardware interface 116 (if present) can be used to operate and control other components of the transmitter component 102. The local computer 106 is also shown to include a local memory 118. The local memory 118 is intended to represent various types of memories or storage devices accessible to the local computing system 110.

[0115] The remote computer 108 is shown to include a remote computing system 120. The remote computing system 120 is intended to represent one or more computing systems or computer cores. The remote computing system 120 is shown to be connected to an optional user interface 124 and a remote network interface 122. The local network interface 112 and the remote network interface 122 are used to form a network connection 114. The remote computing system 120 is shown to be further connected to a remote memory 126. The remote memory 126 is intended to represent various types of memories and storage devices accessible to the remote computing system 120. The local memory 118 and the remote memory 126 can be examples of non-transitory storage media, for example.

[0116] The local memory 118 is shown to include local machine-executable instructions 130. The local machine-executable instructions contain instructions that enable the local computing system 110 to perform basic data analysis and data processing tasks. The local memory 118 is also shown to include a first component 132 of a feature vector, a second component 134 of the feature vector, and a third component 136 of the feature vector.

[0117] The remote memory 126 is shown as including a first component of the feature vector and a second component 134 of the feature vector. The remote memory 126 is also shown as including remote machine-executable instructions 140. The remote machine-executable instructions 140 enable the remote computing system 120 to perform basic data processing and data manipulation tasks. The remote memory 126 is also shown as including a decoder neural network 142, which is configured to output an approximation of the medical image data in response to receiving at least the components 132, 134, 136 of the feature vector. The memory 126 is shown as including an incomplete feature vector 144 that has been assembled from the first component 132 of the feature vector and the second component 134 of the feature vector. The remote memory 126 is also shown as including an approximation 146 of the medical image data received from the decoder neural network 142 in response to the input incomplete feature vector 144.

[0118] Figure 2 A flowchart of a method of the medical system 100 illustrating the operation is shown. First, in step 200, the feature vectors 132, 134, 136 describing the medical image data are received. Next, in step 202, the components 132, 134, 136 of the feature vector are transmitted to the remote computing system 120 via the network connection 114. In this example, the feature vector has been divided into three components 132, 134, and 136. Assigning different parts of the feature vector to these different components 132, 134, 136 is equivalent to assigning an ordering that assigns importance to the various elements of the feature vector. For example, the elements of the feature vector in the first component 132 are considered the most important or relevant parts of the feature vector. The second component 134 contains the second most important component, and the third component 136 contains the third most important component. Figure 1 The division of the feature vector into three components 132, 134, 136 is arbitrary. There can be any number of components into which the feature vector is divided. In some examples, the feature vector can be accompanied by an additional vector that assigns a number or ordering to each individual element. In this particular example, the elements of the feature vector in the first component 132 are transmitted first. It can be seen that in this example, only the first component 132 and the second component 134 have been transmitted to the receiver component 104.

[0119]

[0120] ​After performing step 202, the method proceeds to step 204, where the components 132, 134 of the feature vector are received by the remote computing system 120 via the network interface 114. Then, in step 206, the received components 132, 134 of the feature vector are assembled into a portion of the feature vector. In this case, this portion of the feature vector is the incomplete feature vector 144. Next, in step 208, an approximation 146 of the medical image data is received in response to inputting this portion 144 of the feature vector into the decoder neural network 142.

[0121] Figure 3 Another example of a medical system 300 is illustrated. Figure 3 The depicted medical system 300 is similar to Figure 1 the depicted medical system 100. In medical system 300, the components 132, 134, 136 of the feature vector are stored in different storage locations 302, 304, 306. In the figure, it can be seen that the first component 132 of the feature vector is stored in the first storage location 302. The second component 134 of the feature vector is stored in the second storage location 304. The third component 136 of the feature vector is stored in the third storage location 306. The first storage location 302 has the lowest latency, and the third storage location 306 has the longest latency. The second storage location 304 has a latency intermediate between the latencies of the first storage location 302 and the third storage location 306. This arrangement can be useful in several cases. Since it is desirable to transmit the most important elements of the feature vector first, the lower latency of the first storage location 302 can enable a faster transmission of the first component 132 of the feature vector.

[0122] The components of the feature vector can thus be retrieved and then transmitted in order of importance. In this example, once the component 132 of the feature vector is received, it is automatically transmitted by the local computing system 110 via the network interface 114 to the remote computing system 120. Another case where this can be useful is that in cloud and other types of commercially available storage devices, the latency of the storage location can determine the price. If this is for example Figure 3 the case illustrated, the most important components of the medical image data are quickly available in the first storage location 302. In some applications, it may not be necessary to provide the third component 136 of the feature vector or even the second component 134 of the feature vector. Thus, this can achieve cost savings for data structures such as images or the raw data used to reconstruct medical images. The data structures can be very large, and it may be desirable to make only their components quickly accessible.

[0123] Figure 4 Another example of a medical system 400 is shown. Medical system 400 is similar toFigure 1 The medical system depicted in Figure 1 differs in that it additionally includes a medical imaging system 402. The medical imaging system 402 is intended to represent various types of local anatomical medical imaging systems. The devices and methods depicted herein are particularly applicable to computed tomography.

[0124] The object 404 is shown as leaning on the object support 406. The components of the object 404 are shown as being within the imaging region 408 of the medical imaging system 402. The local memory 118 is shown as additionally containing medical image data 410. In this case, it is the raw measurement data acquired by the medical imaging system 402. The measurement data can be, for example, k-space data from a magnetic resonance imaging system. In other examples, the data is a CT profile from a CT system or an image from a CBCT system.

[0125] The local memory 118 is also shown as containing an encoder neural network 412. The local memory 118 is also shown as containing a feature vector 414 received in response to inputting the medical image data 410 into the encoder neural network 412. In this case, the feature vector 414 has been partitioned into a first component 132 of the feature vector, a second component 134 of the feature vector, and a third component 136 of the feature vector. This can be done, for example, using an importance map neural network or by taking the elements of the feature vector with the largest magnitudes.

[0126] In this example, again only the first component 132 of the feature vector and the second component 134 of the feature vector have been transmitted to the remote computing system 120. As in the previous example, this has been used to provide an approximation 146 of the medical imaging data. In this particular example, it is an approximation of the measurement data acquired by the medical imaging system 402. Then, the memory 126 is also shown as containing a preliminary medical image 416 that has been reconstructed based on the approximation 146 of the medical image data.

[0127] Figure 5 An illustrated operation is shown Figure 4Flowchart of a method of a medical system 400. First, in step 500, medical image data 410 is received. In this case, this can include controlling a medical imaging system 402 to acquire medical image data or measurement data. Next, in step 502, a feature vector 414 is received in response to inputting the medical image data 410 into an encoder neural network 412. Next, in step 504, an ordering of the elements of the feature vector 414 is determined and assigned. In this particular example, this is done by binning the different elements of the feature vector 414 into a first component 132, a second component 134, or a third component 136 of the feature vector. After performing step 504, steps 200, 202, 204, 206, and 208 are performed, as Figure 2 illustrated.

[0128] Figure 6 Illustrates the functionality or inoperability of an encoder neural network 412 and a decoder neural network 142. In this example, they are arranged together as a functional autoencoder 600. In many cases, the encoder neural network 412 and the decoder neural network 142 will be constructed by taking a standard autoencoder neural network and then separating them into two separate neural networks after training has been completed. In this example, medical imaging data 410 is shown as being input into the encoder neural network 412, which then produces a feature vector 414. The feature vector 414 is essentially a latent space vector for the autoencoder 600. When the feature vector 414 is input into the decoder neural network 142, an approximation 146 of the medical image data is output. During training, elements of the feature vector 414 can be discarded to train the decoder neural network 142 to be able to reconstruct an approximation 146 of the medical image data using only components or missing elements of the feature vector 414.

[0129] Figure 7 Shows Figure 6 a modification of the arrangement of the neural network illustrated. In Figure 7 the example shown, the autoencoder structure 600 is still shown. However, in this case, the medical image data 410 is put into an importance map neural network 700, which outputs an ordering 702 of the various elements of the feature vector 414. This ordering 702 can be used to rank or assign an order to individual elements. For example, the ordering 702 can be used to divide the feature vector 414 into a first component 132, a second component 134, and a third component 136, as illustrated in the previous example. The importance map neural network can be formed, for example, by several convolutional or fully connected layers connected to an output layer. The output layer is configured to assign an ordering to each element of the feature vector.

[0130] Figure 8 shows Figure 7 a modification of the arrangement of the neural network illustrated. In Figure 7 , there is a fully separate neural network 700 for providing sorting 702. In this example, there is an importance map neural network layer 800 connected to a layer within the decoder neural network 412. For example, there may be a number of convolutional layers for forming the importance map neural network layer 800, which are connected before the output layer of the encoder neural network 412. Figure 8 The arrangement in shows an alternative way of incorporating the generation of sorting 702 into the encoder neural network 412.

[0131] Although the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.

[0132] Those skilled in the art can understand and realize other variations of the disclosed embodiments when practicing the claimed invention by studying the drawings, the disclosure, and the claims. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. Although specific measures are recited in mutually different dependent claims, this does not indicate that a combination of these measures cannot be used advantageously. A computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium provided together with or as part of other hardware, but the computer program may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

[0133] The invention also relates to a computer program comprising instructions which, when run on a computer, cause the computer to perform any of the above method steps according to the invention.

[0134] The invention also relates to a computer-readable medium comprising instructions which, when run on a computer, cause the computer to perform any of the above method steps according to the invention.

Claims

1. A medical system (100, 300, 400) comprising: A local memory (118) that stores local machine-executable instructions (130), and A local computing system (110), wherein the running of the machine-executable instructions further causes the computing system to: Receive (200) a feature vector (414) that describes medical image data (410), wherein the feature vector is configured to be input into a decoder neural network (142), the decoder neural network being a decoder component of an autoencoder, wherein the decoder neural network is configured to output an approximation (146) of the medical image data when receiving at least a portion of the feature vector as input, wherein the feature vector includes a ranking (702) that assigns importance to elements of the feature vector; and Sequentially transmit (202) components (132, 134, 136) of the feature vector to a remote computing system (120) via a network connection (114), wherein the components of the feature vector having higher importance are transmitted first, wherein the local memory further includes an encoder neural network (412), the encoder neural network being an encoder component of the autoencoder; Wherein the running of the local machine-executable instructions further causes the local computing system to: Receive (500) the medical image data; Receive (502) the feature vector in response to inputting the medical image data into the encoder neural network; and Determine (504) the ranking and assign the ranking to the elements of the feature vector.

2. The medical system according to claim 1, wherein, The determining of the ranking includes: Determining the magnitudes of the elements of the feature vector; and Assigning a ranking to the elements of the feature vector, wherein the ranking increases as the magnitude increases.

3. The medical system according to claim 1, wherein, The local memory further includes an importance map neural network (700) configured to: receive the medical image data as input and, in response, provide as output an assignment that assigns the ranking to the elements of the feature vector, wherein the running of the local machine-executable instructions further includes: receiving from the importance map neural network the ranking of the elements of the feature vector in response to inputting the medical image data into the importance map neural network.

4. The medical system according to claim 3, wherein, The importance map neural network is implemented as any one of: incorporated into the encoder neural network (800), and as a separate neural network (700).

5. The medical system according to any one of the preceding claims, wherein, The ranking includes an array of values that assigns the ranking to the elements of the feature vector.

6. The medical system according to any one of the preceding claims, wherein, The ranking includes: assigning the elements of the feature vector to a discrete plurality of storage locations (302, 304, 306), wherein the discrete plurality of storage locations have different data retrieval latencies, and wherein a lower data retrieval latency is equivalent to a higher ranking.

7. The medical system according to claim 6, wherein, Receiving the feature vector by retrieving the components of the feature vector from the discrete plurality of storage locations according to the sorting.

8. The medical system according to any one of the preceding claims, wherein, The medical system further includes: A remote memory (126) that stores remote machine-executable instructions (140) and the decoder neural network; and A remote computing system (120), wherein the running of the remote machine-executable instructions causes the remote computing system to: Sequentially receive (204) the components of the feature vector via the network connection; Assemble (206) the received components of the feature vector into the part of the feature vector; and Receive (208) an approximation (146) of the medical image data in response to: Inputting the part of the feature vector into the decoder neural network, and wherein the received components of the feature vector are input into the decoder neural network before all components of the feature vector have been received.

9. The medical system according to claim 8, wherein, The medical image data is measurement data, and wherein the running of the remote machine-executable instructions further causes the remote computing system to reconstruct a preliminary medical image (416) based on the medical image data before all components of the feature vector have been received.

10. The medical system according to claim 8, wherein, The medical image data is a medical image, and wherein the running of the machine-executable instructions further causes the computing system to present the approximation of the medical image data.

11. The medical system according to claim 10, wherein, Before the receipt of all components of the feature vector at the remote computing system, the assembly of the received components of the feature vector, the approximation of the medical image data, and the presentation of the approximation of the medical image data are repeated a plurality of times, wherein the running of the remote machine-executable instructions further causes the remote computing system to: Receive a stop transmission instruction in response to the presentation of the approximation of the medical image data; and Transmit the stop transmission instruction to the local computing system via the network connection; Wherein the running of the local executable machine-executable instructions further causes the local computing system to: Receive the stop transmission instruction via the network connection; and Stop transmitting the components of the feature vector in response to receiving the stop transmission instruction.

12. A medical system, comprising: A remote memory (126) that stores remote machine-executable instructions and a decoder neural network (142), the decoder neural network being a decoder component of an autoencoder, wherein the decoder neural network is configured to output an approximation of the medical image data in response to receiving at least a part of a feature vector describing the medical image data as input, wherein the feature vector includes a sorting (702) that assigns importance to elements of the feature vector and the feature vector, and the sorting is determined in response to inputting the medical image data into an encoder neural network in a local computing system (110), the encoder neural network being an encoder component of the autoencoder; and Remote computing system (120), wherein the running of the machine-executable instructions causes the remote computing system to: Sequentially receive (204) the components (132, 134, 136) of the feature vector via the network connection; Assemble (206) the received components of the feature vector into the portion (144) of the feature vector; Receive (208) an approximation (146) of the medical image data in response to: Input the portion of the feature vector into the decoder neural network of the autoencoder, and wherein the portion of the feature vector is input into the decoder neural network before all components of the feature vector have been received.

13. A computer program comprising local machine-executable instructions (130) for execution by a local computing system (110), wherein, The running of the machine-executable instructions also causes the local computing system to: Receive (500) medical image data; Receive (502) a feature vector in response to inputting the medical image data into an encoder neural network, the encoder neural network being an encoder component of an autoencoder; Determine (504) a sorting (702) and assign the sorting to the elements of the feature vector; Receive (200) the feature vector (132, 134, 136) describing the medical image data, wherein the feature vector is configured to be input into a decoder neural network (142), the decoder neural network being a decoder component of the autoencoder, wherein the decoder neural network is configured to output an approximation (146) of the medical image data when receiving at least a portion of the feature vector as input, wherein the feature vector includes the sorting (702) that assigns importance to the elements of the feature vector; and Sequentially transmit (202) the components of the feature vector to the remote computing system (120) via a network connection (114), wherein the components of the feature vector having higher importance are transmitted first.

14. A medical imaging method, wherein, The method includes: Receive (500) medical image data; Receive (502) a feature vector in response to inputting the medical image data into an encoder neural network, the encoder neural network being an encoder component of an autoencoder; Determine (504) a sorting (702) and assign the sorting to the elements of the feature vector; Receive (200) by a local computing system (110) the feature vector (414) describing the medical image data (410), wherein the feature vector is configured to be input into a decoder neural network (142), the decoder neural network being a decoder component of the autoencoder, wherein the decoder neural network is configured to output an approximation (146) of the medical image data when receiving at least a portion of the feature vector as input, wherein the feature vector includes the sorting (702) that assigns importance to the elements of the feature vector; and The local computing system sequentially transmits (202) the components (132, 134, 136) of the feature vector via a network connection (114) to a remote computing system (120), where the components with higher importance in the feature vector are transmitted first. The remote computing system sequentially receives (204) the components of the feature vector via the network connection; The remote computing system assembles (206) the received components of the feature vector into the part (144) of the feature vector; and Receives (208) the approximation of the medical image data in response to the following operations: The remote computing system inputs the part of the feature vector into the decoder neural network, and the received components of the feature vector are input into the decoder neural network before all components of the feature vector have been received.

Images