Image processing apparatus

Abstract

To display data generated by using a machine learning model so as to be visually discriminated.SOLUTION: An image processing apparatus according to an embodiment includes a reception unit, a generation unit, an image processing unit, and a display unit. The reception unit receives a medical image as an input. The processing circuitry generates a generated image having the same visual appearance as a reference image by applying a function having a plurality of predetermined learnable coefficients to the medical image. The image processing unit generates an output image by performing image processing on the generated image to make the generated image visually different from a reference image. The display unit displays the output image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The embodiments disclosed in this specification and the drawings relate to an image processing device. [Background technology]

[0002] With the development of generative AI technology using highly trained machine learning models, it has become possible to artificially generate highly accurate images, making it difficult to distinguish them from actual photographs. For this reason, there is a demand for a system that allows users to intuitively recognize that an image is artificially generated. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 060468 Summary of the Invention [Problem to be solved by the invention]

[0004] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to display data in a manner that makes it visually recognizable that the data was generated using a machine learning model. The problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of the configurations described in the embodiments below can also be considered as other problems. [Means for solving the problem]

[0005] An image processing device according to an embodiment includes a receiving unit, a generating unit, an image processing unit, and a display unit. The receiving unit receives a medical image as input. The generating unit applies a function having a predetermined number of learnable coefficients to the medical image to generate a generated image that has the same appearance as a reference image. The image processing unit performs image processing on the generated image to make it appear visually different from the reference image, thereby generating an output image. The display unit displays the output image. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an image processing apparatus according to the first embodiment. [Figure 2] FIG. 2 is a flowchart illustrating a processing procedure of generated image display processing by the image processing device according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing an example of a display screen when a post-contrast image actually captured by the image processing apparatus according to the first embodiment exists. [Figure 4] FIG. 4 is a diagram showing an example of a display screen when there is no post-contrast image actually captured by the image processing apparatus according to the first embodiment. [Figure 5] FIG. 5 is a diagram showing an example of a display screen when an operation to display an artificial image using a generation AI is performed by the image processing device according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a display screen when an operation to display an artificial image using a generation AI is performed by the image processing device according to the first modified example. [Figure 7] FIG. 7 is a diagram showing an example of a display screen when an operation to display an artificial image using a generation AI is performed by the image processing device according to the second modified example. [Figure 8] FIG. 8 is a diagram showing an example of a display screen when an operation to display an artificial image using a generation AI is performed by an image processing device according to a fourth modified example. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, an embodiment of an image processing apparatus will be described in detail with reference to the drawings. In the following description, components having substantially the same functions and configurations are designated by the same reference numerals, and redundant description will be given only when necessary.

[0008] (First embodiment) FIG. 1 is a diagram showing the configuration of an image processing device 10. The image processing device 10 is connected to external systems such as a Hospital Information System (HIS) and a Radiological Information System (RIS) and external devices such as a medical image diagnostic device via a network. The image processing device 10 can transmit and receive information such as images to and from the external systems and devices via the network. The medical image diagnostic device is a modality that captures images of the inside of a subject, such as an X-ray computed tomography (CT) device, a magnetic resonance imaging (MRI) device, an ultrasound diagnostic device, or an X-ray diagnostic device. The network is, for example, a local area network (LAN). Note that connection to the network may be either a wired connection or a wireless connection. Furthermore, the connection line is not limited to a LAN as long as security is ensured by a virtual private network (VPN) or the like. Connection to a public communication line such as the Internet may also be possible.

[0009] The image processing device 10 includes a memory 11, a communication interface 12, a display 13, an input interface 14, and a processing circuit 15. Although the image processing device 10 will be described below as a single device that executes multiple functions, the multiple functions may be executed by separate devices. For example, the functions executed by the image processing device 10 may be distributed and installed on different console devices or workstation devices.

[0010] The memory 11 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an integrated circuit that stores various types of information. The memory 11 may also be a portable storage medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a flash memory, in addition to an HDD or SSD. The memory 11 may also be a drive device that reads and writes various types of information from and to semiconductor memory elements such as flash memory and RAM (Random Access Memory). The storage area of ​​the memory 11 may be located within the image processing device 10, or may be located in an external storage device connected via a network.

[0011] The memory 11 stores programs executed by the processing circuit 15, various data used in the processing of the processing circuit 15, and the like. As the programs, for example, programs that are installed in advance on a computer from a network or a non-transitory computer-readable storage medium and cause the computer to realize each function of the processing circuit 15 are used. The memory 11 also stores images, medical data, and the like used in various processes. The memory 11 also stores a machine learning model 111, which will be described later. Details of the machine learning model 111 will be described later. Note that the various data handled in this specification are typically digital data. The memory 11 is an example of a storage unit.

[0012] The communication interface 12 is a network interface that controls transmission of communications with external systems and external devices via a network.

[0013] The display 13 displays various types of information. For example, the display 13 outputs medical information generated by the processing circuitry 15, a GUI (Graphical User Interface) for receiving various operations from an operator, etc. For example, the display 13 is a liquid crystal display or a CRT (Cathode Ray Tube) display. The display 13 is an example of a display unit.

[0014] The input interface 14 accepts various input operations from the operator, converts the accepted input operations into electrical signals, and outputs them to the processing circuitry 15. For example, the input interface 14 accepts input of medical information, input of various command signals, etc. from the operator. The input interface 14 is realized by a mouse, keyboard, trackball, switch buttons, a touch screen integrating a display screen and a touchpad, a non-contact input circuit using an optical sensor, a voice input circuit, etc., for performing various processes in the processing circuitry 15. The input interface 14 is connected to the processing circuitry 15 and converts input operations received from the operator into electrical signals and outputs them to the control circuit. Note that, in this specification, the input interface is not limited to those equipped with physical operating components such as a mouse and keyboard. For example, an electrical signal processing circuit that receives electrical signals corresponding to input operations from an external input device provided separately from the device and outputs the electrical signals to the processing circuitry 15 is also an example of an input interface. The input interface 14 is an example of an input unit.

[0015] The processing circuitry 15 controls the overall operation of the image processing device 10. The processing circuitry 15 is a processor that executes a receiving function 151, a generating function 152, an image processing function 153, and a display control function 154 by calling and executing programs in the memory 11.

[0016] The term "processor" used in the above description refers to a circuit such as a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD)), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA). When the processor is a CPU, for example, the processor realizes its function by reading and executing a program stored in a memory circuit. On the other hand, when the processor is an ASIC, for example, the program is not stored in a memory circuit, but the function is directly incorporated into the processor circuit as a logic circuit. Note that each processor in this embodiment is not limited to being configured as a single circuit, but may be configured as a single processor by combining multiple independent circuits to realize its function. Furthermore, multiple components in FIG. 1 may be integrated into a single processor to realize its function. The above description of "processor" also applies to the following embodiments and modifications.

[0017] 1, the receiving function 151, the generating function 152, the image processing function 153, and the display control function 154 are described as being realized by a single processing circuit 15. However, the processing circuit may be configured by combining multiple independent processors, and each processor may execute a program to realize each function. Furthermore, the receiving function 151, the generating function 152, the image processing function 153, and the display control function 154 may each be implemented as an individual hardware circuit. The above description of each function executed by the processing circuit 15 also applies to the following embodiments and modifications.

[0018] Although the image processing device 10 is described as a single console that executes multiple functions, the multiple functions may be executed by separate devices. For example, the functions of the processing circuitry 15 may be distributed and installed in different devices.

[0019] The processing circuitry 15 receives a medical image as an input through a receiving function 151. The processing circuitry 15 that realizes the receiving function 151 is an example of a receiving unit. The medical image is, for example, an image that is actually captured using a medical image diagnostic device. As a medical image, an image that looks natural, does not have any visual unnaturalness, and does not give an intuitive sense of discomfort to a user is generally used.

[0020] The medical image is, for example, a morphological image actually captured using an MRI device, such as a FLAIR (Fluid Attenuated Inversion Recovery) image, a T1-weighted image, or a T2-weighted image. The medical image may be another image captured using an MRI device, or may be an image actually captured by a medical imaging diagnostic device, such as an X-ray CT image or an ultrasound image. The medical image may also be an image acquired using a medical device other than a medical imaging diagnostic device.

[0021] The processing circuitry 15 generates a generated image using the machine learning model 111 based on the medical image using the generation function 152. Hereinafter, the generated image output from the machine learning model 111 will be referred to as an artificial image. The processing circuitry 15 that realizes the generation function 152 is an example of a generation unit. The machine learning model 111 is stored in the memory 11, for example.

[0022] The machine learning model 111 is a function having a plurality of predetermined coefficients that can be learned. The machine learning model 111 is a highly trained trained model. The machine learning model 111 is trained to accept input of a medical image and output an artificial image generated based on the medical image. For example, a highly trained machine learning model such as a generative adversarial network (GAN), a diffusion model, a variational autoencoder, or a flow-based model can be used as the machine learning model 111. A highly trained trained model is also called a generative AI. An artificial image output from the machine learning model 111, such as a generative AI, is an image that looks visually the same as a reference image. The reference image is an image that was actually captured. Furthermore, the reference image is an image that is intuitively recognized as an image that was actually captured. In other words, the reference image is an image that is intuitively recognized as not an image that was generated without actually being captured. Furthermore, the reference image is an image that looks natural and does not look strange as an image used in normal image diagnosis. The reference image is an image in which parameters such as contrast, color tone, sharpness, smoothness, and resolution are the same as those used in normal image diagnosis, and the shape of the displayed organ is displayed in the same display format as in normal image diagnosis. Note that the reference image does not need to be actually used in the process of generating an artificial image using machine learning model 111, as long as machine learning model 111 is trained to generate an artificial image that looks visually the same as the reference image.

[0023] An artificial image has the same visual appearance as an actually captured reference image, and therefore, like a reference image, is an image that can be intuitively recognized as having been actually captured. In other words, an artificial image is an image that can be intuitively recognized as not being an image generated without actually being captured. Furthermore, an artificial image is an image that looks natural and does not feel unnatural as an image used in normal image diagnosis. For this reason, an artificial image is difficult to visually distinguish from an actually captured image, and it is difficult to intuitively recognize that it is an artificially generated image.

[0024] For example, the machine learning model 111 is a generation AI trained to receive an input of a pre-contrast T1-weighted image and generate a post-contrast T1-weighted image. The machine learning model 111 may also be a generation AI trained to receive an input of an image captured by a medical image diagnostic device and predict and generate an image after a certain period of time has elapsed. The machine learning model 111 may also be a generation AI trained to receive an input of a non-contrast T1-weighted image and generate a non-contrast T2-weighted image. The machine learning model 111 may also be a generation AI trained to receive an input of multiple types of images and generate one image. In this way, there are no limitations on the types or numbers of medical images input to the machine learning model 111 and the artificial images generated, and the machine learning model 111 may be any generation AI trained to generate an artificial image using a medical image.

[0025] The processing circuit 15 generates an output image by performing image processing on the artificial image using the image processing function 153 to make the artificial image appear visually different from the reference image. Hereinafter, image processing to make the artificial image appear visually different from the reference image is referred to as processing. The reference image is used as a reference when determining the processing content of the processing. Furthermore, since the artificial image has the same visual appearance as the reference image, processing to make the artificial image appear visually different from the reference image includes image processing to make the artificial image appear visually different from the reference image. The processing circuit 15 that realizes the image processing function 153 is an example of an image processing unit.

[0026] The output image has a visually different appearance from the reference image that was actually captured, and therefore, unlike the reference image, is an image that is intuitively recognized as not being an image that was actually captured. In other words, the output image is an image that is intuitively recognized as an image that was generated without actually being captured. Furthermore, the output image has an unnatural appearance, and is an image that would be unnatural for an image used in normal image diagnosis.

[0027] The output image is an image in which at least one of parameters such as contrast, color tone, sharpness, smoothness, resolution, and display form is significantly different from that of the reference image. In other words, the output image is an image in which at least one of the parameters is significantly different from the value used in normal image diagnosis. Examples of processing that can be used include image processing such as processing that changes the contrast, color tone, sharpness, smoothness, resolution, etc. of the artificial image, and morphological processing that deforms the shape of organs included in the artificial image. The processing may be, for example, processing using a filter.

[0028] For example, when changing the contrast, the processing is a process of significantly increasing or decreasing the contrast. By significantly increasing or decreasing the contrast, the visual appearance of the output image differs from that of the reference image, and the output image can appear unnatural.

[0029] When changing the color tone, the processing is image processing that changes the color tone of the artificial image to cool colors such as blue or green. If the reference image is a monochrome image, changing the color tone of the artificial image to a cool color tone that is not commonly used can make the image visually different from the reference image, and can make the output image look unnatural.

[0030] When changing the sharpness, the processing can be a process that significantly increases the sharpness (a process that significantly sharpens the image) or a process that significantly decreases the sharpness (a process that significantly blurs the image). By significantly increasing or decreasing the sharpness, the image can be made to look visually different from the reference image, and the output image can look unnatural.

[0031] Furthermore, when the smoothness is changed, the processing is a process of significantly increasing the smoothness (excessive denoising) or significantly decreasing the smoothness (excessive noise addition). By significantly increasing the smoothness (making the image excessively smooth) or significantly decreasing the smoothness, the artificial image can be made to look visually different from the reference image, and the output image can look unnatural.

[0032] In addition, when changing the resolution, the processing can be a process of significantly increasing or decreasing the resolution. By significantly increasing the resolution (for example, increasing it by 10 times) or significantly decreasing it (for example, decreasing it to one-tenth the resolution), the output image can be made to look visually different from the reference image, and the output image can look unnatural.

[0033] Furthermore, when the shape of an organ included in an artificial image is deformed, the processing includes a morphological processing that deforms the normal shape of the organ into a shape that is easier to diagnose, a subtraction processing that subtracts a medical image from the artificial image to generate a difference image, and a processing that enhances and superimposes the difference image generated by the subtraction processing on the medical image. By performing the above processing on the artificial image, the output image can be made to look visually different from the reference image, and the parts corresponding to the artificial image can be made to look unnatural.

[0034] The processing is not limited to the above examples, and may be any image processing that makes the displayed portion of the subject appear unnatural. Other image processing that is not generally used in image diagnosis may also be used as the processing.

[0035] Furthermore, the above-mentioned processing is image processing that changes the appearance of the display area of ​​the photographed subject in the artificial image. Therefore, the processing does not include processing that maintains the appearance of the display area of ​​the photographed subject, such as processing that displays text on an image. The output image generated by the above-mentioned processing has a visual unnaturalness, and is an image in which the display area of ​​the photographed subject has been processed. In other words, the output image is an image that has been processed to have a visually different appearance from an image that has a natural appearance. The output image may also be called a processed image.

[0036] The processing circuitry 15 causes various information to be displayed on the display 13 by the display control function 154. For example, the processing circuitry 15 causes the display 13 to display an image, information about the patient in the image, and an interface for performing various operations.

[0037] Furthermore, the processing circuitry 15 causes the display 13 to display an output image using the display control function 154. The processing circuitry 15 that realizes the display control function 154 is an example of a display control unit. For example, in the display control function 154, the processing circuitry 15 causes the display 13 to display a display screen that displays a medical image and an output image generated using the medical image side by side. In the display control function 154, the processing circuitry 15 may display the medical image and the output image in separate windows, or may display only the output image on the display 13.

[0038] (Artificial image display processing) Next, the operation of the artificial image display processing executed by the image processing device 10 will be described. FIG. 2 is a flowchart showing an example of the procedure of the artificial image display processing. Here, as an example, a case will be described in which a generation AI trained to receive an input of a pre-contrast anatomical image and generate a post-contrast anatomical image is used as the machine learning model 111, and subtraction processing is used as the processing. Note that the above machine learning model 111 is just an example, and the machine learning model 111 may be any generation AI trained to generate an artificial image using a medical image. For example, the machine learning model 111 may be a generation AI trained to receive an input of a non-contrast T1-weighted image and generate a non-contrast T2-weighted image, or may be a generation AI trained to receive an input of multiple types of images and generate a single image.

[0039] The processing procedures for each process described below are merely examples, and each process can be modified as appropriate as possible. Furthermore, steps can be omitted, replaced, or added as appropriate for the processing procedures described below depending on the embodiment.

[0040] (Step S101) In the artificial image display process, the processing circuitry 15 first receives a medical image using the receiving function 151. For example, the processing circuitry 15 acquires a pre-contrast anatomical image (hereinafter referred to as a pre-contrast image) as the medical image from the memory 11. The pre-contrast image may be acquired from an external system or may be input by a user.

[0041] (Step S102) Next, the processing circuitry 15 causes the display control function 154 to display the medical image on the display 13. For example, the processing circuitry 15 causes the display 13 to display a display screen showing the acquired pre-contrast image. At this time, if a post-contrast morphological image (hereinafter also referred to as a real post-contrast image) generated by actually capturing an image of the same patient as the pre-contrast image exists in the memory 11, the processing circuitry 15 displays the pre-contrast image and the real post-contrast image side by side. FIG. 3 is a diagram showing an example of a display screen 20 displayed on the display 13 when a real post-contrast image exists. As shown in FIG. 3, a pre-contrast image 21 and a real post-contrast image 22 are displayed side by side on the display screen 20. The real post-contrast image 22 may be displayed in a window separate from the window displaying the pre-contrast image 21, or only the real post-contrast image 22 may be displayed on the display 13.

[0042] On the other hand, if a real post-contrast image corresponding to the pre-contrast image does not exist in the memory 11, the processing circuitry 15 displays only the pre-contrast image. Fig. 4 is a diagram showing an example of a display screen 20 displayed on the display 13 when a real post-contrast image does not exist. As shown in Fig. 4, only a pre-contrast image 21 is displayed on the display screen 20. Furthermore, if a real post-contrast image does not exist, in addition to the pre-contrast image 21, an operation button 23 for artificially generating a post-contrast morphological image using a machine learning model 111 is displayed on the display screen 20.

[0043] (Step S103) The processing circuit 15 continuously detects whether or not an operation input has been made through the operation button 23, and waits until an operation input has been made through the operation button 23 (step S103-No).

[0044] (Step S104) When the user selects the operation button 23 (step S103—Yes), the processing circuitry 15 determines that an instruction to generate an artificial image using the machine learning model 111 has been input. Thereafter, the processing circuitry 15 uses the generation function 152 to input a medical image to the machine learning model 111 to generate an artificial image. For example, the processing circuitry 15 inputs a pre-contrast image to the machine learning model 111. The machine learning model 111 accepts the input of the pre-contrast image, artificially generates a post-contrast morphological image, and generates the artificially generated post-contrast morphological image (hereinafter referred to as a post-artificial contrast image). The post-artificial contrast image is an image that appears visually identical to an actually captured image, such as the pre-contrast image 21 or the actual post-contrast image 22. In other words, the post-artificial contrast image is an image that appears visually identical to a reference image. The processing circuitry 15 acquires the post-artificial contrast image generated by the machine learning model 111, associates it with the pre-contrast image, and stores it in the memory 11.

[0045] (Step S105) Next, the processing circuitry 15 performs processing on the artificial image using the image processing function 153. For example, the processing circuitry 15 performs subtraction processing on the acquired post-artificial contrast image. The processing circuitry 15 generates a difference image as an output image by subtracting the luminance value of the pre-contrast image from the luminance value of the post-artificial contrast image.

[0046] (Step S106) Next, the processing circuitry 15 causes the display control function 154 to display the output image on the display 13. For example, the processing circuitry 15 causes the display 13 to display the generated output image alongside the pre-contrast image. FIG. 5 is a diagram illustrating an example of a display screen 20 displayed on the display 13. As illustrated in FIG. 5, a difference image 24 is displayed as an output image next to a pre-contrast image 21 on the display screen 20. The difference image 24 is an image that appears visually different from actually captured images such as the pre-contrast image 21 and the actual post-contrast image 22. That is, the difference image 24 is an image that appears visually different from the reference image. Therefore, the difference image 24 also appears visually different from the post-artificial contrast image, which appears visually the same as the reference image. By confirming that the difference image 24, which appears unnatural, is displayed instead of the post-contrast image, which appears natural, such as the actual post-contrast image 22 in FIG. 3, the user can intuitively recognize that the displayed post-contrast image is an artificially generated image. The difference image 24 may be displayed in a window separate from the window displaying the pre-contrast image 21 , or only the difference image 24 may be displayed on the display 13 .

[0047] The effects of the image processing device 10 according to this embodiment will be described below.

[0048] The image processing device 10 according to this embodiment receives a medical image. The medical image is, for example, the pre-contrast image 21 shown in FIGS. 3 to 5. The image processing device 10 generates an artificial image based on the medical image using a machine learning model 111. The machine learning model 111 is a function having a plurality of predetermined coefficients that can be trained. The artificial image is an image generated using a highly trained machine learning model 111 such as a generative AI. With the recent technological level of generative AI, an artificial image generated using a highly trained machine learning model 111 looks natural, making it difficult to visually distinguish it from an actually captured image.

[0049] The image processing device 10 then performs processing on the artificial image to make it look visually different from artificial images or medical images, thereby generating an output image, and displays the generated output image on the display 13. The processing is image processing that makes the artificial image look visually unnatural. For example, the processing is subtraction processing that subtracts the pre-contrast image 21 of Figures 3 to 5 from the post-artificial contrast image. Due to the processing, the output image has a visually unnatural look compared to the artificial image or medical image. For example, the difference image 24 of Figure 5 can be used as the output image.

[0050] 5 has been subjected to the subtraction process, the difference image 24 looks visually different from the actual post-contrast image 22 that would be generated if the same patient as the pre-contrast image 21 were actually imaged. That is, the output image looks visually different from the image that would be generated if the same subject as the artificial image were actually imaged.

[0051] 2 to 5, the pre-contrast image 21, the real post-contrast image 22, and the post-artificial contrast image generated by the machine learning model 111 are examples of reference images that look natural. That is, the processing is image processing that makes the image look visually different from the reference image, in other words, image processing that makes the reference image look visually unnatural.

[0052] Furthermore, an actually captured reference image is displayed without any processing, as in the actual post-contrast image 22 in FIG. 3. On the other hand, an artificially generated image is displayed in a processed state as a difference image 24, as in FIG. 5. That is, processing is performed on the artificial image generated using the machine learning model 111, which is not performed on the actually captured image. In this way, the image processing device 10 of this embodiment can use different ranges of parameters that affect the appearance of the image between the actually captured image and the artificially generated image. In this case, the ranges of parameters applied to the actually captured image and the artificially generated image are set to ranges that are not adjacent and are significantly different.

[0053] With the above configuration, the image processing device 10 of this embodiment can impart an unnatural appearance to the artificial image generated by the generation AI by further performing image processing that changes the appearance of the artificial image generated by the machine learning model 111. When a user sees an artificial image generated by the generation AI, the user can visually and intuitively recognize that the artificial image differs from a natural image that was actually captured and generated. In other words, by imparting a sense of incongruity or unnaturalness to the artificial image artificially generated by the generation AI, the user can intuitively determine that the image is an artificial image and not an actually captured image, without having to check the manual.

[0054] There is a technique for displaying text (character indicator) on or near an artificial image generated using a generation AI, indicating that the image is an artificial image. However, with this technique, the display area of ​​the subject of the image does not change, and the appearance of the display area of ​​the subject does not change. This requires the user to carefully search for the text and, if the user focuses on the display area of ​​the subject in the image, the user may miss the text. In this case, the user will not realize that the image is an artificial image. On the other hand, the image processing device 10 according to this embodiment performs image processing on the artificial image generated by the machine learning model 111, modifying the display area of ​​the subject in the artificial image. This changes the appearance of the display area of ​​the subject itself, so that the user can realize that the image they are viewing is an artificial image even if they focus on the display area of ​​the subject in the image.

[0055] 2 to 5, a post-contrast morphological image is artificially generated using the machine learning model 111 when a real post-contrast image corresponding to the pre-contrast image is not present in the memory 11. However, even when a real post-contrast image corresponding to the pre-contrast image is present in the memory 11, a post-artificial contrast image may be generated using the machine learning model 111 and used as a supplement to the real post-contrast image. In this case, the display screen 20 shown in FIG. 3 displays an operation button 23 in addition to the pre-contrast image 21 and the real post-contrast image 22. When the user selects the operation button 23, the processing circuitry 15 inputs the pre-contrast image into the machine learning model 111 to generate a post-artificial contrast image. Thereafter, the processing circuitry 15 displays the pre-contrast image 21, the real post-contrast image 22, and the post-artificial contrast image side by side. By comparing the real post-contrast image 22 with the post-artificial contrast image, the user can confirm whether the real post-contrast image 22 has been captured as expected.

[0056] (First Modification) The first modification will be described below. This modification is obtained by modifying the configuration of the first embodiment as follows.

[0057] In the first embodiment, the processing performed is a subtraction process that removes the medical image from the artificial image generated by the machine learning model 111. In this modification, the processing circuitry 15 in the image processing function 153 performs, as processing, a process of enhancing the difference image generated by the subtraction process that subtracts the medical image from the artificial image to generate a difference image and superimposing the difference image on the medical image.

[0058] As an example, similar to the first embodiment, the operation will be described when a generation AI trained to receive input of a pre-contrast anatomical image and generate a post-contrast anatomical image is used as the machine learning model 111. In this modification, in the process of step S105 in FIG. 2 , the processing circuitry 15 first performs subtraction processing on the acquired post-artificial contrast image using the image processing function 153. At this time, the processing circuitry 15 generates, as an output image, a difference image obtained by subtracting the luminance value of the pre-contrast image from the luminance value of the post-artificial contrast image. Next, the processing circuitry 15 uses the image processing function 153 to enhance the generated difference image and generate a superimposed image by superimposing it on the medical image. At this time, for example, the processing circuitry 15 changes the color or density of the difference image before superimposing it on the medical image.

[0059] Fig. 6 is a diagram showing an example of the display screen 20 displayed on the display 13 in the processing of step S106 in Fig. 2. As shown in Fig. 6, a superimposed image 25 is displayed as an output image next to the pre-contrast image 21 on the display screen 20. The user can visually recognize that the superimposed image 25, which appears unnatural, is displayed instead of a post-contrast image, which appears natural like the actual post-contrast image 22 in Fig. 3, and can intuitively recognize that the displayed post-contrast image is an artificially generated image.

[0060] In this way, in this modification, by further performing image processing that changes the appearance of the artificial image generated by the machine learning model 111, it is possible to impart an unnatural appearance to the artificial image generated by the generation AI, thereby achieving the same effect as in the first embodiment. Furthermore, in this modification, by superimposing the pre-contrast image 21 on the difference image 24 in Fig. 5, it is possible to display the positional relationship on the organ of the part that has changed since pre-contrast administration in an easily understandable manner.

[0061] When the subtraction image is superimposed on a medical image such as the pre-contrast image 21, only the tumor portion in the subtraction image may be extracted and superimposed on the medical image. In this case, only the tumor portion is emphasized and superimposed, so that the tumor portion can be displayed prominently.

[0062] (Second Modification) The second modification will be described below. This modification is obtained by modifying the configuration of the first embodiment as follows.

[0063] In the first embodiment, the processing performed is a subtraction process that removes the medical image from the artificial image generated by the machine learning model 111. In this modification, the processing circuitry 15 in the image processing function 153 performs image processing that changes the contrast of the artificial image as the processing. For example, image processing that inverts the contrast of the artificial image or image processing that changes the contrast of the entire artificial image is performed.

[0064] As an example, similar to the first embodiment, a description will be given of the operation when a generation AI trained to receive input of a pre-contrast anatomical image and generate a post-contrast anatomical image is used as the machine learning model 111. In this modification, in the process of step S105 in Fig. 2, the processing circuitry 15 uses the image processing function 153 to perform a process of inverting black and white of the acquired post-artificial contrast image, and generates an inverted image with inverted black and white as an output image.

[0065] 7 is a diagram showing an example of a display screen 20 displayed on the display 13 in the processing of step S106 in FIG. 2. As shown in FIG. 7, an inverted image 26 is displayed as an output image on the display screen 20 next to the pre-contrast image 21. The user can visually recognize that the unnatural-looking inverted image 26 is displayed, rather than the natural-looking post-contrast image like the actual post-contrast image 22 in FIG. 3, and can intuitively recognize that the displayed post-contrast image is an artificially generated image.

[0066] In this way, in this modified example, by further performing image processing that changes the appearance of the artificial image generated by the machine learning model 111, it is possible to impart an unnatural appearance to the artificial image generated by the generation AI, thereby achieving the same effect as in the first embodiment.

[0067] 7, for example, a generation AI that has been trained to receive an input of a T1-weighted image and generate a T2-weighted image may be used as the machine learning model 111. In this case, for example, a process of changing the contrast may be executed as a processing process on the T2-weighted image artificially generated using the machine learning model 111, and the T2-weighted image with the changed contrast may be generated and displayed as an output image.

[0068] (Third Modification) In the first embodiment, after displaying the pre-contrast image 21, a post-artificial contrast image is generated based on a user's instruction, and the post-artificial contrast image is processed to display a subtraction image 24. In this modification, when the pre-contrast image 21 is displayed, the subtraction image 24 is automatically generated and displayed by processing the post-artificial contrast image.

[0069] Fig. 8 is a diagram showing an example of the display screen 20 that is displayed on the display 13 when there is no actual post-contrast image in this modified example. As shown in Fig. 8, a pre-contrast image 21 and a subtraction image 24 are displayed side by side on the display screen 20, and the operation buttons 23 shown in Figs. 4 to 7 are not displayed.

[0070] Furthermore, the machine learning model 111 may be trained to artificially generate images generated using the same modality as medical images, or may be trained to artificially generate images generated using a modality different from that of medical images.

[0071] According to at least one of the embodiments described above, it is possible to display data in a manner that makes it visually recognizable that the data has been generated using a machine learning model.

[0072] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0073] 10...Image processing device 11...Memory 111...Machine learning model 12...Communication interface 13...Display 14...Input interface 15...Processing circuit 151…Acceptance function 152…Generation function 153...Image processing function 154...Display control function 20…Display screen 21...Pre-contrast image 22...Actual post-contrast image 23...Operation buttons 24...Difference image 25...Superimposed image 26...Inverted image

Claims

1. a receiving unit that receives a medical image as an input; a generation unit that applies a function having a plurality of predetermined coefficients that can be learned to the medical image to generate a generated image that has the same appearance as the reference image; an image processing unit that generates an output image by performing image processing on the generated image to make it appear visually different from the reference image; a display unit that displays the output image; An image processing device comprising:

2. The reference image is an image that has actually been captured. The image processing device according to claim 1 .

3. The image processing is a subtraction process of subtracting the medical image from the generated image. The image processing device according to claim 1 .

4. The image processing is a process of enhancing a difference image generated by subtraction processing of subtracting the medical image from the generated image and superimposing the difference image on the medical image. The image processing device according to claim 1 .

5. The image processing is a process of changing the contrast of the generated image. The image processing device according to claim 1 .

6. the image processing is filtering; The image processing device according to claim 1 .

7. The image processing is a process of changing the color tone of the generated image. The image processing device according to claim 1 .

8. The image processing is a process of changing a display area of ​​the photographed subject in the generated image. The image processing device according to claim 1 .

Images