Apparatus for providing immediate advice regarding selections at the time of imaging for rationalizing an imaging workflow

Abstract

The imaging system SYS includes a medical imaging device IA. The medical imaging device comprises a detector D for acquiring a first image of a patient in an imaging session and a display unit DD for displaying the first image on a screen. The system further includes a mobile image processing device MIP distinct from the medical imaging device IA. The mobile image processing device MIP comprises an interface IN for receiving a representation of the first image and an image analyzer IAZ configured to analyze the representation and, based on the analysis, calculate medical decision support information during the imaging session. The decision support information is displayed on an on-board display device MD of the mobile image processing device MIP.

Description

Technical Field

[0001] The present invention relates to an image processing system, the use of a mobile image processing device in such a system, a mobile image processing device, a method of image processing, a computer program element and a computer-readable medium.

Background Art

[0002] Previously, medical imaging equipment was mainly operated by specialized operators such as radiologic technologists (X-ray, CT or MRI), sonographers (ultrasound) or nuclear medicine technologists (NM imaging). However, there is a new trend that less capable staff are taking on the examinations. Such practices, without protective measures, can potentially lead to a decline in clinical quality.

[0003] The operator (referred to herein as the "user") is responsible for performing a series of work steps throughout the examination, depending on the type and details of the equipment, for example: (i) positioning the patient; (ii) adjusting the parameters of the imaging scan during the procedure; (iii) performing the acquisition itself; and (iv) reviewing and post-processing the resulting images at the console of the imaging device; throughout the entire examination.

[0004] When the imaging examination is completed, the subsequent steps in a modern radiology workflow typically involve the operator electronically transmitting the images to an image database (PACS) for storage and simultaneously transmitting them via an interpretation worklist to other trained specialists (medically certified radiologists) for interpretation of the findings of the examination. Depending on multiple factors such as the urgency of the medical situation and the organization of the workload specific to the medical institution, this interpretation is often performed asynchronously, meaning that there is a significant time delay between image acquisition and image interpretation.

[0005] Artificial intelligence (AI) has the potential to complement the shortage of qualified personnel while also improving clinical efficiency. An AI system is a system implemented on a computer. These systems are based on machine learning algorithms that are pre-trained with training data to perform tasks such as assisting the user during an examination. Such AI systems exist, but they are typically integrated with a given imaging device or hospital IT equipment for a given medical facility. Furthermore, these AI systems can vary from facility to facility and may not be easy to operate, or the AI output may not always be easily understandable. Additionally, some medical facilities, such as those in rural areas or emerging markets, may simply not have such AI systems at all.

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0006] Therefore, there may be a need for systems and methods that address at least some of the above-described deficiencies.

MEANS FOR SOLVING THE PROBLEMS

[0007] The object of the present invention is solved by the subject matter of the independent claims, and further embodiments are included in the dependent claims. It should be noted that the aspects of the imaging (image processing) system according to the present invention described below equally apply to the use of a mobile image processing device in the system, to the mobile processing device, to a method of image processing, to a computer program element, and to a computer-readable medium.

[0008] According to a first aspect of the present invention, there is provided an imaging (image processing) system, the imaging system comprising: a medical imaging device (also referred to herein as an "imager") having a detector for acquiring a first image of a patient in an imaging session and a display unit for displaying the first image on a screen; and a mobile image processing device different from the medical imaging device; and the mobile image processing apparatus comprises: an interface for receiving the representation of the first image; an image analyzer configured to analyze the representation and calculate medical judgment support information based on the analysis during the imaging session; and an on-board display device for displaying the judgment support information; and has.

[0009] The mobile image processing apparatus ( "MIP") is preferably separate and independent from the medical imaging device. The interface is general-purpose and provides interoperability with a series of different medical imaging devices even in different formats. The interface is not incorporated into the imaging device (apparatus), and thus the mobile device is independent in the sense that it can interface with any imager. The MIP can be used as an add-on to an existing imaging device. The MIP can be used at the time of imaging. Specifically, the analyzer is configured to calculate the judgment support information ( "DSI") in real time, i.e., during the imaging session. The imaging session includes the period during which the patient is within or at the imaging device, or at least the period during which the patient is within the examination room where the imaging device is present.

[0010] In an embodiment, the interface of the mobile image processing apparatus includes an imaging element configured to capture the first image displayed during the imaging session as a second image, and the second image forms the representation.

[0011] In other words, this embodiment is based on the direct imaging ( "image of an image") of the displayed image. In other embodiments, the interface is configured as NCF or Bluetooth (registered trademark) if the imaging device is so equipped. Other embodiments include LAN, WLAN, etc. as before.

[0012] In an embodiment, the decision support information includes one or more of: i) a recommended workflow related to the patient, ii) instruction information regarding the image quality related to the first image, iii) instruction information regarding medical findings, and iv) priority information.

[0013] In an embodiment, the recommended workflow is different from a previously defined workflow assumed for the patient.

[0014] In an embodiment, the instruction information regarding the image quality includes instruction information for any one or more of: a) the patient's placement, b) the collimator settings, c) the contrast, d) the resolution, e) the noise, and f) the artifacts.

[0015] In an embodiment, the image analyzer includes a machine learning element that has been pre-trained.

[0016] In an embodiment, the recommended workflow is implemented automatically or after receiving a user's command via the user interface of the mobile device.

[0017] In an embodiment, the image analyzer is fully integrated into the mobile device, or at least a part of the image analyzer is integrated into a remote device that can be communicably coupled to the mobile device via a communication network.

[0018] In an embodiment, the mobile image processing device is a handheld device including any one of: i) a mobile phone, ii) a laptop computer device, and iii) a tablet computer.

[0019] In another aspect, there is provided a mobile image processing device when used in the system according to any one of the above embodiments.

[0020] In another aspect, there is provided the use of the mobile image processing device in a system according to any one of the above embodiments.

[0021] In other aspects, there is provided a mobile image processing apparatus including an imaging element capable of acquiring an image representing medical information related to a patient and including analyzer logic configured to calculate judgment support information related to the patient based on the image. The imaging element includes an image recognition module that cooperates with an autofocus module of the imaging element, and the recognition module is configured to recognize at least one rectangular object within the field of view of the imaging element.

[0022] In an embodiment, the analyzer logic is implemented, for example, in a processor circuit configured for parallel computing such as a multi-core processor, a GPU, or a part thereof.

[0023] The image analyzer may be included in a system-on-chip (SoC) circuit.

[0024] In other aspects, there is provided a method of image processing, the method comprising: acquiring, by a detector of a medical imaging device, a first image of a patient in an image session; displaying the first image on a screen; receiving, by a mobile image processing device different from the medical imaging device, a representation of the first image; analyzing the representation and calculating medical judgment support information based on the analysis during the imaging session; and displaying the judgment support information on an on-board display device. having.

[0025] In other aspects, there is provided a computer program element adapted to cause a processing unit to execute the method when executed by at least one processing unit.

[0026] In other aspects, there is provided a computer-readable medium storing the program element.

[0027] As used herein, the "user" is a medical staff member who is at least partially involved in the imaging procedure in an administrative or organizational manner.

[0028] The "patient" is the person being imaged, or in the case of a veterinarian, an animal (especially a mammal).

[0029] A "machine learning ("ML") element" is any computing unit or device that implements an ML algorithm. The ML algorithm can learn from examples (the "training data"). The performance of the ML element on a task that can be measured by a performance metric, i.e., learning, is usually improved by the training data. Some ML algorithms are based on an ML model adapted based on the training data.

Brief Description of the Drawings

[0030]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

DETAILED DESCRIPTION OF THE INVENTION

[0031] Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings (not to scale).

[0032] Referring to FIG. 1, this figure shows a schematic block diagram of a configured AR assumed in medical or clinical facilities. However, the following description is not necessarily limited to the medical field.

[0033] In a GP general clinic, clinic, hospital, or other medical facility, patient PAT undergoes procedures at the reception desk CD. The patient PAT already has a treatment plan PL assigned, or a treatment plan PL is assigned at the reception desk CD. The treatment plan PL defines a plurality of medical procedures to be performed on the patient. One step of such a procedure may include imaging for diagnostic or treatment purposes. The imaging can be performed by an imaging device IA.

[0034] The imaging device IA can be of any type such as transmission or radiation imaging. Transmission imaging includes, for example, X-ray based imaging performed using a CT scanner or others. Magnetic resonance imaging MRI is also envisioned, and ultrasonic imaging is also envisioned. Radiation imaging includes PET / SPECT and other nuclear medicine modalities. To perform the imaging, the patient PAT is guided to an imaging room IR (see FIG. 4) where the imaging device IA is located.

[0035] During the imaging session, an image IM of the patient is requested. The image IM is preferably in digital format and can assist the doctor in making a diagnosis. To facilitate correct imaging during the imaging session, the configuration includes a computerized system SYS for supporting the imaging operation of the imaging device IA. The user US1 does not necessarily have to be a doctor with a medical degree and can instead be a medical technician or a user with little training. The system SYS encourages the safe and correct use of the imaging device (imager) even for staff with a low level of medical skills, semi-skilled or in on-the-job training, etc.

[0036] The system SYS preferably includes a mobile image processing device MID, which can be operated by the user US1 to assist in the task of correctly and safely acquiring an image of the patient PAT during the imaging session. The device MID, referred to herein as the "mobile device" MID, is a separate entity different from the imaging device IA. As will be shown in more detail below, the mobile device MID includes a general-purpose interface IF that can receive a copy IM' of the image acquired by the imaging device, referred to herein as the "source image" IM.

[0037] The mobile device MID particularly includes an image analyzer element IAZ that enables the copy image IM' to be analyzed to obtain judgment support information that can be displayed on the on-board display OD of the mobile device MID. This information can, for example, assist the user US1 in evaluating whether the source image IM is of sufficient quality. The displayed information can include proposals for further steps and can include a proposal for re-imaging if it is found that the quality is poor. Additionally or alternatively, the information can indicate the presence of a medical condition and can further include a proposal to change a pre-assigned plan PL. Based on the analysis performed by the mobile device MID, the plan PL can be adapted or changed as will be explained in more detail below.

[0038] In accordance with the above-described judgment support information, user US1 can decide to transfer the source image IM to an image storage unit such as a PACS via the hospital communication network CN. The hospital information equipment HIS may include other databases DB, servers SV, or other workstations WS2 of other users US2 that can be accessed via the communication network CN. In addition to or instead of transferring the source image to the storage unit, the source image can also be directly transferred to doctor US2 at workstation WS2, for example, for interpretation or "reading" to confirm a diagnosis. As another example, the doctor can also retrieve the image from the PACS. As described above, technician US1 usually does not participate in the interpretation of images. This task is entrusted to doctor US2 who has a medical degree with training in image interpretation. The user US1 of the imaging device IA can be assisted by the mobile device MID and can focus his / her attention only on the technical considerations for correctly acquiring a source image IM of sufficient quality in accordance with the correct protocol. In this way, doctor US2 can be assured that the correct image has been acquired, and the doctor can direct his / her attention to the interpretation of the image and will not be bothered by the technical aspects of image acquisition.

[0039] Here, looking more closely at the assumed configuration AR, continuing to refer to FIG. 1, the imaging device IA generally includes a signal source SS. During image acquisition in an imaging session, the signal source SS emits an interrogation signal that interacts with the patient's tissue. As a result of the interaction with the tissue, the signal is changed. The changed signal is then detected by the detector unit D. The acquisition circuit converts the detected signal, such as intensity, into a digital image, i.e., the source image IM.

[0040] Adjustment and overall control of the imaging parameters of the imaging device across the entire image acquisition are executed by a technical user US1 from an operator console OC that may include a stationary computing device. The operator console OC can be placed in the same room IR as the imaging apparatus IA, or may be placed in another room. This operator console is communicably coupled to a display device (referred to herein as monitor MD) associated with the operator console OC and the imaging apparatus IA. The acquired image is transferred by the acquisition circuit to a computing unit (workstation) WS1 within the operator console OC operable by the user US1. The operator console can be communicably coupled to the HIS via the network CN.

[0041] The acquired source image IM can be displayed on the main monitor MD. This enables the user US1 to roughly confirm whether the source image is correct. Previously, when the user US1 felt that the image was correct, the source image or a plurality of source images (video) acquired in time series could be transferred into the hospital's information structure via a communication network towards an intended destination such as a PACS, or could be transferred directly to the doctor US2 at the doctor's workstation WS2.

[0042] As proposed herein, before user US1 decides to transfer source image IM to hospital equipment, user US1 can analyze the source image using mobile device MID to determine image quality and / or medical findings. The analysis is performed by the mobile device MID obtaining a copy IM' of the source image IM and then analyzing the copy image IM'. Advantageously, as proposed herein, the mobile device MID is not integrated or "bundled" with the hospital information structure or with the imaging device IA, operator console or workstation. Rather, the mobile image processing device MID is a separate, independent stand-alone unit that is preferably capable of analyzing the received copy IM' itself to calculate judgment information and displaying the sympathy information on its own display OD for user US1. This is advantageous because not all medical facilities have a quality assessment function for imaging available at the time of imaging. Specifically, in a given imaging device in a given department or facility, it is not known whether the quality assessment function is integrated with the workstation WS1 or the operator console. User US1 may be itinerant, i.e., assigned to different departments of the same medical hospital or assigned to work at different medical facilities in different geographical regions, and thus may be required to operate a series of different medical imaging devices spanning different manufacturers and / or different modalities. In such a situation, user US1 can consistently use their mobile device MID to analyze the acquired images with high reliability regardless of the specific equipment. This ensures consistent medical quality across facilities.

[0043] Next, referring to the block diagram of FIG. 2, this figure provides further details of the assumed mobile image processing device MID. As described above, the mobile device MID includes a general-purpose interface IN that enables it to receive copy IM' regardless of the specific imaging equipment.

[0044] In one embodiment, the general-purpose interface IN can be configured as a camera equipped with an image sensor S. The mobile device MID can be configured as a smartphone, tablet, laptop, notebook, or any other computing device equipped with an integrated camera.

[0045] The mobile device MID has its own on-board display (display device) OD. The acquired copy IM' can be displayed on this display as needed. Additionally or alternatively, the judgment information provided by the image analyzer IAZ can be displayed on the on-board display device OD.

[0046] The image analyzer IAZ can be driven by artificial intelligence. In particular, the image analyzer IAZ can be included as a pre-trained machine learning element or model. The image analyzer IAZ can be executed in the processing unit of the mobile device MID. The processing unit can include a general-purpose circuit and / or a dedicated computing circuit such as a GPU, or can be a dedicated core of a multi-core multiprocessor. Preferably, the processing unit is configured for parallel computing. This configuration is particularly advantageous when the underlying machine learning model is a neural network such as a convolutional network. Such types of machine learning models can be efficiently implemented by multiplying vectors, matrices, or tensors. Such types of calculations can be accelerated in parallel computing facilities.

[0047] The mobile device MID can further have a communication device including a transmitter TX and a receiver RX. The communication device enables connection to the hospital network CN. The assumed communication capabilities include Wi-Fi (registered trademark), wireless communication, Bluetooth (registered trademark), NFC, or any one or more of the others.

[0048] In a preferred embodiment, the mobile device is configured to perform an "image of an image" function for obtaining a copy IM' of the source image IM. More specifically, after the source IM is acquired and displayed on the main display (main monitor) MD, the user US1 operates the mobile device MID to capture the image of the source image IM displayed on the main display MD. The image captured in this way forms the copy image IM'.

[0049] To better assist the user US1 in capturing this copy image IM', the image sensor S can be coupled to an autofocus AF function that automatically adjusts the focus and / or exposure. More preferably, the autofocus AF is coupled to an image recognition module IRM that assists the user US1 in capturing the copy image IM' with good focus on the source image IM displayed on the main monitor MD. For this purpose, the image recognition module IRM is configured to search for a field of view of a square or rectangular object. This is because such a field of view is the expected shape of the source image when displayed on the main monitor MD or the shape of the main display MD itself. During focus adjustment by automatic object shape recognition, the contour of the captured object can be shown within the field of view to assist the user US1. For example, a square or rectangular contour representing the boundary of the main display MD represented within the current field of view, or the boundary of the source image IM currently displayed on the main display can be visualized.

[0050] Once the correct object is in focus, the user requests image capture by operating a virtual or physical shutter button UI. The captured image, i.e., the copy IM', is stored in the internal memory of the mobile device MID. The captured copy image IM' is transferred to an image analyzer IAZ for analysis. To exclude irrelevant information, the captured image is automatically trimmed before analysis so that the remaining pixel information represents only the medical information from the source image IM.

[0051] The resolution of the copy IM' is generally lower than that of the source image IM and is determined by the resolution capabilities of the image sensor S. To appropriately account for this reduced resolution, the mobile device may include a settings menu that allows the user to input the original resolution of the source image. The resolution capabilities of the sensor, and thus the resolution of the image copy IM', can be obtained automatically or supplied by the user. Based on this data, i.e., the two resolutions or their ratio, the image analyzer IAZ can take the reduced resolution into account when analyzing the copy image IM'.

[0052] Other settings that can be specified by the user may include the purpose of the imaging, in particular, the specification of an anatomical structure of interest such as the chest, head, arm, leg, or abdomen. The user may also be able to input specific general patient characteristics of the patient, such as gender, age, weight, if possible. Preferably, the on-board display is configured to accept touch screen input. A user interface UI, such as a graphical UI, can be displayed on the on-board screen, and the user can apply or access the settings via the UI.

[0053] The image analyzer IAZ preferably analyzes the image in two stages. In the first stage, the image quality such as resolution, correct collimator settings (if any), etc. is determined. The contrast of the image can also be analyzed. If the image quality meets certain predetermined criteria, the image can be further analyzed to determine the medical condition. If a medical condition is found, this medical condition can preferably be flagged on the on-board display OD with a priority level. The priority level may include a designation of "low", "medium" or "high" priority and / or the name of the medical condition. Alternatively, a finer or coarser grading of the priority levels can also be used. For example, if the presence of an infectious disease such as tuberculosis is confirmed, this can be flagged as a case of high urgency. If no medical condition is found, confirmation information such as "OK" can be displayed, or simply no display. Additionally or alternatively, indication information regarding the image quality is displayed to show the user whether the current image quality IQ meets the predetermined IQ criteria. The predetermined IQ criteria can be set by the user.

[0054] Therefore, the decision support information calculated by the image analyzer may include any one or more of IQ, medical findings and / or related priority levels. Additionally or alternatively, if a medical condition is found, a related workflow can be proposed and displayed. This proposed workflow may be different from the currently assigned plan PL. If the user accepts the change to the proposed workflow, the user can operate the user interface UI to start and register the changed plan PL. This can be achieved by the mobile device connecting to the network CN and sending an appropriate message to the reception desk CD or the responsible doctor US2, etc. If it is found by IAZ that the IQ is insufficient, reshooting (re-imaging) can be proposed, optionally accompanied by a proposal of updated imaging parameters. The user US1 can then accept the reshooting using the UI, and an appropriately formatted message is sent to the operator console OC to adjust the imaging parameters and / or start the re-acquisition of the image.

[0055] The above-described functions of the mobile device MID can be implemented by installing software on a general-purpose handheld device equipped with imaging capabilities. This can be achieved by the user US1 downloading an "app" from a distribution server, the "app store" to their general-purpose handheld device.

[0056] To better assist the user US1 in capturing a copy image IM' of the source image, a positioning device PD can be provided for the mobile device MID, as described below with reference to the embodiments of FIGS. 4 and 5A to 5D. However, such a positioning device PD is optional, and the user can alternatively simply hold the device in front of the main screen MD when capturing the image IM', as shown in the schematic usage example in FIG. 3.

[0057] First, referring to FIG. 4, the figure shows another positioning device PD that enables the user to arrange the mobile device MID horizontally beside the main display. In this way, the positioning device includes a cradle for receiving the mobile device with a clip or mounting means, and the cradle can be attached to, for example, the side edge or upper edge of the main monitor MD by the clip or mounting means. Therefore, the user US1 can easily operate the mobile device MID and the console CO hands-free and clearly view the main display MD and the on-board display OD of the mobile device MID.

[0058] Next, referring to FIG. 5A, this figure shows another embodiment of the positioning device PD in a plan view. This embodiment may include an arm having a clip or other attachment means at one end thereof. The arm can be attached to the edge of the main monitor MD via the above attachment means. The positioning device PD terminates at the other end with a preferably articulated cradle for receiving the imaging device MID. Using such a positioning device enables a hands-free operation for the user, and image acquisition can be activated by voice recognition when the user makes a predefined utterance such as "capture" to operate the mobile device MID to capture an image in the current field of view. The image analyzer may include logic that takes into account the angular deviation α expected when the mobile device captures an image not from the front but at an angle α. The above angle can be adjusted thanks to the joints of the cradle.

[0059] Preferably, the camera device is fully integrated into the mobile device, but this is not necessarily the case in all embodiments. In that case, as shown in FIG. 5B, there is an external camera device XC communicably coupled to the mobile device MID via Bluetooth (registered trademark) or any other optional wireless or wired communication means. In this embodiment, the external camera can be attached to the user's forehead via the headband PD. This configuration enables capturing an image not obliquely as in FIG. 4A but completely from the front. Also in this case, the image acquisition of the copy IM' can be started by a voice command or by the user using an actual or virtual shutter button provided by the mobile device MID. As another example, although not shown, the external camera XC can also be placed on a small tripod in front of the monitor that is properly aligned.

[0060] The embodiment of the positioning device PD in FIG. 5C also enables capturing an image from the front. In this embodiment, this is achieved by using a neckband or strap around the user's neck, with the mobile device hanging from the neckband or strap on the connector. In this case, the mobile device during use is placed on the user US1's chest, and can be made to obtain a front image, especially when using the front camera (if any) of the device MID. In contrast to the rear camera, the front (selfie) camera is a camera that can capture an image of an object with the user interface of the device MID or the on-board display OD facing the object.

[0061] In another embodiment according to FIG. 5D, a periscopic adapter PA attached to the viewfinder of the integrated camera of the mobile device MID is provided. The attachment can be made, for example, via a suction cup. The periscopic adapter can direct the optical path obliquely. During imaging, the mobile device can lie flat on a surface such as a shelf of an operator console or a workbench with the viewfinder facing up.

[0062] Next, referring to FIG. 6, this figure shows an example of how the mobile device can be used in a hospital's information technology facility. The image analyzer IAZ can be fully integrated within the mobile device MID, but alternative embodiments are also envisioned where at least a part or all of the image analysis capabilities are outsourced to a "smart engine" SE that is placed as a function in one of the servers SV of the communication network CN or in a remote server that is not part of the network but can be connected to the network. For example, after installing the aforementioned app, the user can purchase a subscription to access the cloud-based image analyzer function.

[0063] Although the mobile device MID itself is independent of specific hospital equipment or imaging apparatus IA, a certain level of integration is possible via standard interfaces such as Bluetooth (registered trademark), LAN, WLAN, etc. Thus, based on the received judgment support information, the user can directly request from the mobile device the transfer of the source image IM via the hospital network of the PACS, to other users US, etc.

[0064] Referring further to FIG. 6, in an embodiment, according to the priority assigned to the analyzed copy image IM', a plurality of different reading (decoding) queues (waiting queues) RQ and RQ - can be determined. Then, the corresponding source image IM is split into these queues. Specifically, the source image given a higher priority than others based on the analysis of the corresponding copy image is transferred to the higher-priority reading queue RQ, while the less urgent image is demoted to the second image queue RQ - of less urgent images. Thereby, the image reader (decoder) US2 can better manage his / her workload.

[0065] Specifically, based on the analysis of the copy image by the smart engine, the corresponding source image IM is transmitted (routed) from the imaging apparatus IA to the PACS via the network CN. This transmission can be requested by the user from the mobile device MID, or can also be requested from the workstation WS1 or the console OC. The smart engine SE analyzes the image and transfers judgment support information to the proposed device MID. Then, the user US1 can, via the confirmation feedback from the device MID, use the appropriate AE (application entity) title to direct the source image from the imaging apparatus to the PACS and transfer it into the corresponding queues RQ and RQ - and permit it.

[0066] The above-mentioned smart engine may include software elements that operate on appropriate hardware within the local IT facility SV. The network connection to the proposed device MID can be implemented using a LAN, WLAN, or others as required. In one embodiment, there is a feedback communication channel that enables the radiologist US2 to provide image quality feedback during image interpretation that may significantly occur after actual image acquisition.

[0067] The above-mentioned feedback information and / or decision support information can be collected and stored as statistical information in the same or another database QS. The statistical information STAT represents an overall picture of the IQ (image quality) of the images generated in the relevant medical facility or a group of such facilities. This aspect is further shown in FIG. 7. FIG. 7 shows an overview of the integration of the smart engine and the image quality statistics database for the purpose of retrospective analysis of the image quality state over a specified period. FIG. 7 shows how the proposed device MID is integrated into a larger system for image quality monitoring, for example, enabling retrospective analysis of the image quality state by the management radiology staff. Such an evaluation can be carried out not only as a baseline evaluation at the start of a quality improvement initiative but also for continuously monitoring the image quality. Images are retrieved from the PACS, and the quality measurements performed by the smart engine are saved in the quality statistics database. The intermediate results of the statistical analysis can be automatically transferred to the mobile device MID once, periodically, or upon the user's request and displayed on the on-board display OD. A web server can be used to host the smart engine together with a database management system for the statistical data STAT.

[0068] Figure 8 is a schematic diagram of a network integrated with a smart engine in a user-adaptive training situation. The picture quality information and related statistical information STAT are used for the purpose of retrospectively analyzing the picture quality state over a specified period. User-adaptive training can be carried out. The analysis of picture quality statistics can identify individual training recommendations for a specific user US1, and this can be deployed through the recommended system. The quality statistics database QS hosted by the smart engine server is connected to user-specific training content TD. User US1 can use a standard office PC and, in an embodiment, can start a client such as a web-based client to access the adjusted content TD. The mobile device MID can be used as an application for accessing the training content together with the above client. The executed training session is saved in the training record database together with the results. The system has a training user interface that enables it to retrieve any one or more of recommendations (e.g., from a supervisor or a more experienced colleague considering user-specific statistics), a training framework, and training content. In an embodiment, a web / client-based reporting application can be used to access this information. The above training content can be saved on the smart engine SE. The content can be customized, for example, by an administrator.

[0069] FIG. 9 shows an overview of the network integrated with the smart engine for deploying the clinical decision support system. The proposed device MID is used to display the results of the analysis of the image IM (e.g., transmitted via LAN) or the copy image IM' through a clinical decision support application that can be executed by the smart engine. Specifically, the proposed device MID can be used to display the results of clinical decision support at the time of imaging. The copy image IM' or the acquired source image IM is transmitted to the smart engine server SE and analyzed by the clinical decision support application. Immediate feedback is sent to the mobile device MID regarding high-priority images HP for which particularly immediate workflow steps are required to draw the attention of the user US1. For example, if an infectious disease is detected in the image, the patient must be immediately isolated from other patients in the hospital to prevent the spread of the infection. Other low-priority images LP are transferred to the PACS and stored in the appropriate folder (AE title).

[0070] It will be understood that the principles of the embodiments of FIGS. 6 to 9, such as decoding queues and statistical evaluations, can also be implemented in embodiments without a remote smart engine, that is, in embodiments where the image analyzer is implemented wholly or partially in the mobile device MID itself.

[0071] Refer to FIG. 10, which shows a flowchart of the method of image processing related to the system described above. However, it will be understood that the method described below is not necessarily restricted to the system described above. Therefore, the following method can be understood as a teaching in itself.

[0072] In step S1010, a first digital image of a patient (referred to herein as a source image) is acquired by an imaging device in an imaging session.

[0073] In optional step S1020, the source image is displayed on the static screen of the first display unit.

[0074] In step S1030, the second digital representation (the "copy" image) of the source image is received by the image processing device. The image processing device is preferably a mobile type such as a handheld device, and is a separate one independent of a stationary computing unit such as a workstation and / or an operator console coupled to the medical imaging device.

[0075] In step S1040, this second image, i.e., the copy image, is analyzed to calculate medical support information related to the source image during the imaging session.

[0076] In step S1050, the calculated medical judgment support information is displayed on the on-board display device of the mobile processing device.

[0077] In optional step S1060, a user response is received via the user interface of the mobile device. The user response represents an action requested in relation to the displayed judgment support information. The user can, for example, request one or more of the workflow steps proposed to be executed in relation to the patient. The requested workflow step (or steps), which can be different from the pre-assigned workflow, can include re-imaging of the image, consultation with an expert, or reservation of other medical devices in the current or other medical facilities.

[0078] In further step S1070, the user's request is initiated by sending a corresponding message via the network to the recipient, for example, to a reception desk CD or a device associated with a doctor.

[0079] As another example, one or more of the recommended work steps are automatically executed without user confirmation via the interface. In this embodiment, when the copy image is analyzed, a modified workflow is initiated by sending a corresponding message or control signal to related network actors including the imaging machine IA, hospital IT facilities, etc.

[0080] In an embodiment, the copy image is captured by an imaging element (component) of a mobile device. The copy image is an "image of an image", in other words, an image representation of a source image acquired by the imaging element while the source image is displayed on a main display device associated with the imaging device.

[0081] The imaging element is preferably integrated into the mobile imaging device, but alternatively, an external imaging element connectable to the mobile device can also be used. Instead of this "image of an image" method, a copy of the source image can also be transferred to the mobile imaging device via interface means such as attachment to NFC, Wi-Fi (registered trademark), email or text message, or by Bluetooth (registered trademark) transmission.

[0082] The calculated judgment support information includes one or more of a recommended workflow related to the patient, indication information of the image quality of the source image, indication information of medical findings related to the patient such as the medical condition, and preferably related priority information. The above priority information represents the urgency of the medical findings.

[0083] Preferably, the calculation of the judgment support information is performed in a two-stage sequential processing flow. In the first stage, the image quality is determined. Only when it is found that the image quality is sufficient, the image is then analyzed for the proposal of medical findings and / or workflow. The workflow calculated based on the analyzed image may be different from, for example, the workflow originally associated with the patient at the time of reception. This change in the workflow may be required, for example, when an unexpected medical condition that was not previously assumed by the original workflow is detected in the image. For example, when a patient is scheduled to receive cancer treatment for a specific organ such as the liver, a specific workflow is assumed. However, if it happens to be revealed by the analysis of the copy image that the patient actually has pneumonia, it is necessary to change the workflow so that the pneumonia is treated first before proceeding with the cancer treatment.

[0084] For image quality analysis, it may include evaluation of patient positioning, collimator settings (if any), contrast, resolution, image noise, or artifacts. Some or all of these factors may be considered and represented as a single image quality score on an appropriate scale, or each factor may be measured by a separate score on a different scale. If it is found that the image quality is of sufficient quality, in an embodiment, no further display is made on the on-board screen of the mobile device. As another example, preferably, if the image quality is considered sufficient, a suggestive graphic display is shown. For example, a suggestive "tick" symbol may be displayed in an appropriate color scheme such as green. If it is found that the image quality is insufficient, this is also indicated by a suggestive symbol such as a red penalty point on the on-board display. If a medical condition is found, this is indicated by appropriate text or other symbols on the on-board display of the mobile display device. The recommended workflow based on the findings may also be additionally or alternatively displayed.

[0085] In an embodiment, the user interface of the mobile device may be configured to receive user input via the user interface. The workflow that may probably be proposed in response to the user input received in this way may be initiated by sending an appropriate message to the patient registration department CD via the communication network and thereafter. Additionally or alternatively, the message can be sent to a second user US2 such as the attending physician together with the findings, warning the physician to pay attention to the patient.

[0086] Preferably, the decision support information is supplied in real time after the representation of the source image is received by the mobile device. In particular, the result of the analysis, which is the decision support information, shall be made available within seconds or within one of its minutes. The calculations required for the analysis may be fully executed by the processing unit of the mobile device, or may be outsourced partially or entirely to an external remote server with more powerful processing capabilities.

[0087] In an embodiment, the recommended workflow may include a recommendation for re - imaging an image based on an analysis. In this case, technician US1 can decide to follow this advice. Since the judgment - support information is available in real - time, the user can respond immediately to this, and the patient can re - image the image while still in the imaging device or while still in the imaging session. Unnecessary transmission of inadequate images via the network to hospital information facilities such as PACS can be avoided. Thereby, waste of network traffic and memory space can be reduced.

[0088] In an embodiment, the analysis step S1040 is based on a machine - learning model pre - trained. The machine - learning model is pre - trained with past patient data that can be obtained from the image storage section of the same hospital or other hospitals. Preferably, a supervised - learning method in which past images are pre - labeled by experienced clinicians is used. The labeling provides target data including any one or more of instruction information regarding the medical conditions existing in the past images, instruction information regarding the proposed workflow, and instruction information on whether the image quality is considered sufficient.

[0089] The training of the machine - learning element may include, in one or more iterations, the step of receiving training data and the step of applying a machine - learning algorithm to the training data. As a result of this application, a pre - trained model is obtained and can be used for deployment. At the time of deployment, new data (for example, a copy image IM' not from the training set) can be applied to the pre - trained model to obtain the desired judgment - support information regarding this new data.

[0090] The source image to be displayed and captured does not necessarily have to be a single still image, and there can also be a plurality of consecutively displayed source images, that is, a video or a movie. All of the above and below are equally applicable to such videos and movies.

[0091] Referring to FIG. 11 here, a neural network model that can be used in the embodiment is shown in the figure. However, instead of a neural network, other machine learning techniques such as a support vector machine or a decision tree can also be used. That being said, neural networks, especially convolutional networks, have been found to be particularly beneficial, especially for image data.

[0092] Specifically, FIG. 11 is a schematic diagram of a convolutional neural network CNN. The fully configured NN obtained after training (to be described in more detail below) is considered to be a representation of an approximation of the latent mapping between two spaces, one or more images of either an image quality metric, medical findings, and treatment plan, and a space. These spaces can be represented as points in a potentially high-dimensional space where the image is an NxN matrix and N is the number of pixels. The IQ metric, medical findings, and treatment plan can likewise be encoded as vectors, matrices, or tensors. For example, the workflow can be implemented as a matrix or vector structure where each entry represents a step in the workflow. The learning task can be one or more of classification and / or regression. The input space of the image can include a 4D matrix to represent a time series of matrices, i.e., a video sequence.

[0093] A suitable trained machine learning model or element attempts to approximate this mapping. The approximation can be achieved in a learning or training process where the parameters that themselves form a high-dimensional space are adjusted by an optimization method based on the training data.

[0094] More specifically, the machine learning element can be realized as a neural network ("NN"), especially a convolutional neural network ("CNN"). Continuing to refer to FIG. 11, the figure shows in more detail the CNN architecture assumed in the embodiments herein.

[0095] The CNN operates in two modes, namely the "training mode / phase" and the "deployment mode / phase". In the training mode, the initial model of the CNN is trained based on a set of training data to generate a trained CNN model. In the deployment mode, the pre-trained CNN model is supplied with new data that is not trained and operates during normal use. The training mode can be a one-time process, or the mode can be continued in an iterative training phase to improve performance. All of the above described regarding the two modes applies to any kind of machine learning algorithm and is not limited to CNN, or more specifically, NN.

[0096] The CNN has a group of interconnected nodes organized into layers. The CNN includes an output layer OL and an input layer IL. The input layer IL can be a matrix whose size (rows and columns) matches that of the training input image. The output layer OL can be a vector or matrix having a size that matches the size selected for the image quality metric, medical findings, and treatment plan.

[0097] The CNN preferably has a deep learning architecture. That is, there is at least one, preferably two or more hidden layers between OL and IL. The hidden layer can include one or more convolutional layers CL1, CL2 ("CL"), one or more pooling layers PL1, PL2 ("PL"), and / or one or more fully connected layers FL1, FL2 ("FL"). The CL is not fully connected, and / or the connections from the CL to the next layer can be different, but generally, they are fixed in the FL.

[0098] Each node is associated with a number (referred to as "weight") that represents how the node responds to the input from the previous node in the previous layer.

[0099] All combinations of weights define the configuration of the CNN. During the learning phase, the initial configuration is adjusted based on the training data using forward-backward ("FB") propagation or other optimization methods, or learning algorithms such as other gradient descent methods. The gradients are obtained with respect to the parameters of the objective function.

[0100] The training mode is preferably supervised, i.e., based on annotated training data. The annotated training data contains pairs of training data items. For each pair, one item is the training input data and the other item is the target training data that is known in advance to be correctly associated with its own training input data item. This association defines the annotation and is preferably provided by a human expert. The training pairs include past images as training input data, and what is associated with each training image is the target of a label for any one or more of the instruction information of IQ, the instruction information of the medical findings represented by the image, the instruction information of the priority level, and the instruction information of the workflow steps required for a given image.

[0101] In the training mode, preferably, a plurality of such pairs are supplied to the input layer so as to propagate through the CNN until the output appears at OL. Initially, the output generally differs from the target. During optimization, the initial configuration is readjusted to achieve a good match between the input training data and their corresponding targets for all pairs. The said match is measured by a similarity measure that can be formulated with respect to the objective function or cost function. The aim is to adjust the parameters so that they incur a low cost, i.e., result in a good match.

[0102] More specifically, in the NN model, input training data items are supplied to the input layer (IL), passed through a series of successive connections of convolutional layers CL1, CL2 and possibly one or more pooling layers PL1, PL2, and finally passed to one or more fully connected layers. The convolutional module is responsible for feature-based learning (e.g., identifying features of patient characteristics and context data, etc.), while the fully connected layer is responsible for more abstract learning, such as the impact of the above features on treatment, etc. The output layer OL contains output data representing the estimated values of each target.

[0103] The exact grouping and order of the layers according to FIG. 11 are merely exemplary embodiments, and other groupings and orders of the layers are also envisioned in different embodiments. Also, the number of layers of each type (i.e., any of CL, FL, PL) may differ from the configuration shown in FIG. 11. The depth of the CNN may also differ from that shown in FIG. 11. All of the above apply equally to other NNs envisioned herein, such as fully connected classical perceptron type NNs (deep or non-deep) and regression NNs. Different from the above, unsupervised learning or reinforcement learning methods may also be envisioned in different embodiments.

[0104] The annotated (labeled) training data envisioned herein may need to be reformatted into a structured format. As described above, the annotated training data may be arranged as vectors, matrices or tensors (arrays of dimensions greater than 2). This reformatting may be performed by a data preprocessor module (not shown), such as a script program or filter that operates through the patient records of the current facility's HIS to extract a series of patient characteristics.

[0105] The training data set is supplied to the initially configured CNN and processed according to a learning algorithm such as the FB propagation algorithm as described above. At the end of the training phase, the CNN pre-trained in this way can be used in the deployment phase to calculate the decision support information for new data, i.e., newly acquired copy images that do not exist in the training data.

[0106] Some or all of the above-described steps can be implemented in hardware, software, or a combination thereof. Implementing in hardware may include a properly programmed FPGA (Field Programmable Gate Array) or a hard-wired IC chip. For excellent responsiveness and high throughput, a multi-core processor such as a GPU or TPU can be used to implement the above-described training and deployment of machine learning models, especially for NNs.

[0107] One or more features disclosed herein can be configured and implemented as circuits encoded in a computer-readable medium and / or combinations thereof. Circuits can include discrete and / or integrated circuits, application-specific integrated circuits (ASICs), system-on-chips (SOCs), and combinations thereof, as well as machines, computer systems, processors and memories, and computer programs.

[0108] In another exemplary embodiment of the present invention, a computer program or computer program element is provided, and the computer program or computer program element is adapted to execute the method steps of the method according to one of the above embodiments on a suitable system.

[0109] Therefore, the above computer program element can be stored in a computer unit that can also be part of an embodiment of the present invention. This computing (computer) unit can be adapted to execute the steps of the above-described method or induce the execution of the steps. Further, the unit can be adapted to operate the components of the device. The computing unit can be adapted to operate automatically and / or execute user instructions. The computer program can be loaded into the working memory of a data processor. Therefore, the data processor can be equipped to execute the method of the present invention.

[0110] This exemplary embodiment of the present invention covers both a computer program that uses the present invention from the start and a computer program that changes from an existing program to a program that uses the present invention by an update.

[0111] Furthermore, the computer program elements can provide all the necessary steps for fulfilling the procedures of the exemplary embodiments of the methods described above.

[0112] According to a further exemplary embodiment of the present invention, a computer-readable medium such as a CD-ROM is provided, and the computer-readable medium stores the computer program elements described in the above paragraphs.

[0113] The computer program can be stored and / or distributed by a suitable medium (in particular, a non-transitory medium, although not necessarily so), such as an optical storage medium or a solid-state medium, which is supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.

[0114] However, the computer program can also be presented via a network such as the World Wide Web and downloaded from such a network into the working memory of a data processor. According to a further exemplary embodiment of the present invention, a medium for enabling the computer program elements to be downloaded is also provided, and the computer program elements are configured to execute the method according to one of the aforementioned embodiments of the present invention.

[0115] Note that the embodiments of the present invention are described in relation to different subjects. In particular, while some embodiments are described in relation to method-type claims, other embodiments are described in relation to apparatus-type claims. However, those skilled in the art will understand from the above and the following descriptions that, unless otherwise specified, any combination of features belonging to a certain type of subject, as well as any combination between features related to different subjects, are considered to be disclosed by this application. Furthermore, all features can be combined to provide a synergistic effect greater than a mere collection of these features.

[0116] As described above, the present invention has been illustrated and described in detail in the drawings and the above description. However, such illustrations and descriptions should be regarded as explanatory or exemplary and not restrictive. The present invention is not limited to the disclosed embodiments. Other modifications to the disclosed embodiments can be understood and implemented by those skilled in the art from a review of the drawings, the present disclosure, and the dependent claims when implementing the invention described in the claims.

[0117] In the claims, the term "comprising" does not exclude other elements or steps, and the singular form does not exclude the plural. A single processor or other unit can perform the functions of several items described in the claims. The mere fact that certain means are recited in different dependent claims does not indicate that a combination of these means cannot be used advantageously. Any reference signs in the claims should not be construed as limiting the scope. Hereinafter, various embodiments of the present invention will be appended. (Appendix 1) A medical imaging device having a detector for acquiring a first image of a patient in an imaging session and a display unit for displaying the first image on a screen, A mobile image processing device separate from the medical imaging device, An interface for receiving a representation of the first image, An image analyzer that analyzes the representation and calculates medical judgment support information based on the analysis during the imaging session, and An on-board display device for displaying the medical judgment support information A mobile image processing device having, An image processing system having. (Appendix 2) The interface of the mobile image processing device has an imaging element that captures the first image displayed during the imaging session as a second image, and the second image forms the representation. The image processing system according to Appendix 1. (Appendix 3) The judgment support information includes any one or more of i) a recommended workflow for the patient, ii) image quality instruction information regarding the first image, iii) instruction information on medical findings, and iv) priority information. The image processing system according to Appendix 1 or Appendix 2. (Appendix 4) The recommended workflow is different from a previously defined workflow assumed for the patient. The image processing system according to Appendix 3. (Appendix 5) The image quality instruction information includes instruction information on any one or more of a) patient positioning, b) collimator settings, c) contrast, d) resolution, e) noise, and f) artifacts. The image processing system according to Appendix 3. (Appendix 6) The image analyzer includes a pre-trained machine learning element. The image processing system according to any one of Appendices 1 to 5. (Appendix 7) The recommended workflow is performed automatically or after receiving a user's command via the user interface of the mobile image processing device. The image processing system according to Appendix 3. (Appendix 8) The image analyzer is integrated as a whole into the mobile image processing device, or at least a part of the image analyzer is integrated into a remote device that can be communicatively coupled to the mobile image processing device via a communication network. The image processing system according to any one of Appendices 1 to 7. (Appendix 9) The image processing system according to any one of Appendices 1 to 8, wherein the mobile image processing device is a handheld device including any one of i) a mobile phone, ii) a laptop computing device, and iii) a tablet computer. (Appendix 10) A mobile image processing device used in the image processing system according to any one of Appendices 1 to 9. (Appendix 11) A mobile image processing device including an imaging element capable of acquiring an image representing medical information about a patient and including analyzer logic for calculating judgment support information about the patient based on the image, wherein the imaging element includes an image recognition module that cooperates with an autofocus module of the imaging element, and the image recognition module recognizes at least one rectangular object within the field of view of the imaging element. (Appendix 12) The mobile image processing device according to Appendix 11, wherein the analyzer logic is implemented within a processor circuit configured for parallel calculation. (Appendix 13) Obtaining a first image of a patient in an imaging session by a detector of a medical imaging device; Displaying the first image on a screen; Receiving a representation of the first image by a mobile image processing device separate from the medical imaging device; Analyzing the representation and calculating medical judgment support information during the imaging session based on the analysis; And displaying the medical judgment support information on an on-board display device. A method for image processing. (Appendix 14) A computer program that, when executed by at least one processing unit, causes the processing unit to execute the method according to Appendix 13. (Appendix 15) A computer-readable medium storing the computer program according to Appendix 14.

Claims

1. A medical imaging device having a detector for obtaining a first image of a patient in an imaging session and a display unit for displaying the first image on a screen, A mobile image processing device separate from the medical imaging device, An interface for obtaining a representation of the first image, An image analyzer that analyzes the representation of the first image and calculates medical judgment support information based on the analysis during the imaging session, and An on-board display device for displaying the medical judgment support information A mobile image processing device having, having, wherein the interface of the mobile image processing device has an imaging element that captures the first image displayed during the imaging session as a second image, and the second image is a representation of the first image, an image processing system.

2. The image processing system according to claim 1, wherein the medical judgment support information includes any one or more of i) a recommended workflow for the patient, ii) image quality instruction information for the first image, iii) instruction information for medical findings, and iv) priority information.

3. The image processing system according to claim 2, wherein when the medical judgment support information includes the recommended workflow, the recommended workflow is different from a previously defined workflow assumed for the patient.

4. The image processing system according to any one of claims 1 to 3, wherein the image quality instruction information includes instruction information for any one or more of a) patient positioning, b) collimator setting, c) contrast, d) resolution, e) noise, and f) artifacts.

5. The image processing system according to any one of claims 1 to 4, wherein the image analyzer includes a pre-trained machine learning element.

6. The image processing system according to claim 2, claim 3, or claim 4 or claim 5 that directly or indirectly quotes claim 2 or claim 3, wherein the recommended workflow is performed automatically or after receiving a user command via a user interface of the mobile image processing device.

7. The image processing system according to any one of claims 1 to 6, wherein the image analyzer is integrated as a whole into the mobile image processing device, or at least a part of the image analyzer is integrated into a remote device communicatively connectable to the mobile image processing device via a communication network.

8. The image processing system according to any one of claims 1 to 7, wherein the mobile image processing device is a handheld device including any one of i) a mobile phone, ii) a laptop computing device, and iii) a tablet computer.

9. A mobile image processing device used during an imaging session with a medical imaging device, an interface that captures a first image of a patient obtained by the medical imaging device during the imaging session and displayed on a screen of the medical imaging device to obtain a representation of the first image; an image analyzer that analyzes the representation of the first image in real time and calculates medical judgment support information in real time based on the analysis; an on-board display device for displaying the medical judgment support information in real time, and the interface has an imaging element that captures the first image displayed on the screen, the mobile image processing device.

10. The mobile image processing device according to claim 9, wherein the imaging element includes an image recognition module that cooperates with an autofocus module of the imaging element, and the image recognition module recognizes at least one rectangular object within the field of view of the imaging element.

11. The medical imaging device includes the steps of: obtaining a first image of a patient during an imaging session by a detector of the medical imaging device; displaying the first image on a screen during the imaging session by the medical imaging device; a mobile image processing device separate from the medical imaging device capturing the first image to obtain a representation of the first image; the mobile image processing device analyzing the representation of the first image and calculating medical judgment support information during the imaging session based on the analysis; the mobile image processing device displaying the medical judgment support information on an on-board display device and having a method of image processing.

12. A computer program that, when executed by at least one processing unit, causes the processing unit to execute the method according to claim 11.

13. A computer-readable medium storing the computer program according to claim 12.

Images