Image generation device, image generation method, image generation program, and recording medium

Abstract

To improve a feeling of trust in an analysis result obtained by using a machine model.SOLUTION: An image creation device (3) comprises: an acquisition unit (32) that acquires an analysis result (41) output from a machine model (601) for analyzing a medical image obtained by picking up an image of a subject, and attention area information (42) related to an attention area that is an area in the medical image to which attention is paid in a process in which the analysis result is output; and an image creation unit (31) that creates a display image obtained by changing the medical image based on the attention area information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image generation device, an image generation method, an image generation program, and a recording medium.

Background Art

[0002] For example, Patent Document 1 discloses a radiographic age estimation model that shows the correspondence between a feature amount obtained from radiographic image data and the age of a subject by using the radiographic image data of the subject and the information on the age of the subject when the radiographic image was taken.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, various images have been analyzed using a trained machine model. However, the analysis results obtained using the machine model often lack clear basis. The purpose of the present disclosure is to improve the reliability of the analysis results obtained using the machine model.

Means for Solving the Problems

[0005] An image generation device according to one aspect of the present disclosure includes an acquisition unit that acquires an analysis result output from a machine model that analyzes a medical image of a subject, and attention area information regarding an attention area that is an area within the medical image and that was focused on during the process of outputting the analysis result, and an image generation unit that generates a display image obtained by changing the medical image based on the attention area information.

[0006] An image generation method according to one aspect of the present disclosure includes an acquisition step of acquiring analysis results output from a machine model that analyzes medical images of a subject, and focus region information relating to a region within the medical image that was focused on during the process of outputting the analysis results, and an image generation step of generating a display image by modifying the medical image based on the focus region information.

[0007] Each aspect of the present disclosure may be implemented by a computer, in which case a control program for the image generation device that enables the computer to implement the image generation device by operating the computer as each part (software element) of the image generation device, and a computer-readable recording medium on which the program is recorded, also fall within the scope of the present disclosure. [Effects of the Invention]

[0008] According to one aspect of the present invention, the confidence in analysis results obtained using a mechanical model can be further improved. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the configuration of an image generation device according to Embodiment 1 of this disclosure. [Figure 2] This is a flowchart showing the flow of the image display method S1 according to Embodiment 1. [Figure 3] This is a schematic diagram showing an example of a display image according to Embodiment 1. [Figure 4] This is a block diagram showing the configuration of an image remote analysis system according to Embodiment 2 of this disclosure. [Figure 5] This is a schematic diagram showing an example of a browser image for user input. [Figure 6] This is a schematic diagram showing an example of X-ray image data used for bone density analysis. [Figure 7] This is a schematic diagram showing an example of a browser image of the analysis results transmitted by the analysis result transmission unit. [Figure 8]Figure 7 is a schematic diagram showing an example of a browser image with the derivation data added to the displayed image. [Figure 9] Figure 8 is a magnified schematic diagram of the image of region 503. [Figure 10] This is a schematic diagram showing an example of a browser image that displays the current and future risk of fractures in patients. [Figure 11] This is a schematic diagram showing an example of a browser image with the derivation data added to the display image shown in Figure 10. [Figure 12] This is a schematic diagram showing an example of a display image with added data for the derivation of cellular pathology analysis results. [Figure 13] This is a schematic diagram illustrating an example of scrolling to display images on a single screen. [Modes for carrying out the invention]

[0010] [Embodiment 1] Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. Figure 1 is a block diagram showing the configuration of an image generation device 3 according to Embodiment 1 of the present disclosure. The image generation device 3 is a device that generates a display image by modifying a medical image to include the basis for the analysis result, based on focus region information related to a focus region that was identified during the process in which an analysis result is output from a machine model that analyzes a medical image. The image generation device 3 can communicate information with an analysis device 60 and a display device 70 via a communication unit 50. The analysis device 60 includes a machine model 601. The machine model 601 is, for example, a trained machine model that analyzes at least one of an image and a numerical value. The display device 70 displays, for example, the results analyzed by the machine model 601.

[0011] As shown in Figure 1, the image generation device 3 comprises a control unit 30, a storage unit 40, and a communication unit 50. The control unit 30 comprises an image generation unit 31, a data acquisition unit (acquisition unit) 32, and a communication control unit 33.

[0012] The data acquisition unit 32 acquires the analysis result (analysis data) output (derived) from the machine model 601 that analyzes the image, and information regarding the basis for deriving the analysis result (derivation basis data). As an example, the derivation basis data is information on the region of interest in the medical image, which is the region of interest that was focused on during the process of outputting the analysis result. The data acquisition unit 32 records the acquired analysis data 41 and derivation basis data 42 in the storage unit 40.

[0013] The image generation unit 31 generates a display image (display image data) obtained by modifying the medical image based on the region-of-interest information. The image generation unit 31 records the generated display image data 43 in the storage unit 40.

[0014] The communication control unit 33 transmits the display image data 43 recorded in the storage unit 40 to the display device 70. In this case, the communication control unit 33 may transmit the display image data 43 to the display device 70. Alternatively, the communication control unit 33 may first transmit the image data used for the analysis and then transmit the analysis data 41. This can increase the transmission speed.

[0015] The control unit 30 performs overall control of the entire image generation device 3. The control unit 30 includes at least one processor and at least one memory. The processor can be configured using, for example, at least one general-purpose processor such as one or more MPUs (Micro Processing Units) or CPUs (Central Processing Units). The memory may include multiple types of memories such as ROM (Read Only Memory) and RAM (Random Access Memory). As an example, the processor realizes the functions of the image generation device 3 by expanding and executing various control programs recorded in the ROM of the memory in the RAM. Also, the processor can be an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate It may also include a processor configured by an array or a PLD (Programmable Logic Device) or the like.

[0016] FIG. 2 is a flowchart showing the flow of an image display method S1 using an image generation device 3 as an example. As shown in the figure, the image display method S1 includes steps from step S11 to step S13.

[0017] The data acquisition unit 32 acquires an analysis result derived by a machine model 601 that analyzes an image and information regarding the basis for deriving the analysis result (S11). Specifically, for example, the data acquisition unit 32 acquires an analysis result output from a machine model that analyzes a medical image of a subject and focus area information regarding a focus area within the medical image that was the focus during the process of outputting the analysis result. The area within the medical image may be, for example, an area inside the outer periphery of the medical image. The focus area information may be, for example, entirely located within the area of the medical image or only partially overlapping. Or, when the focus area information is not in the medical image (or, for example, when it is the entire medical image), it may not overlap with the area within the medical image. In this case, for example, a message indicating that the focus area does not exist (or, for example, that the entire medical image is the focus area) may be displayed outside the area within the medical image.

[0018] The image generation unit 31 generates a display image for displaying the acquired analysis result and information regarding the basis for deriving the analysis result on one screen (S12). Specifically, the image generation unit 31 generates a display image obtained by modifying the medical image based on the focus area information.

[0019] The communication control unit 33 transmits the display image generated by the image generation unit 31 to the display device 70 (step S13). Through this processing method, the analysis results derived by the machine model that analyzes the image and information regarding the basis for the derivation of the analysis results can be displayed on a single screen. Using this method, as an example, a display image 100 is generated and displayed on the display device 70, which displays on a single screen the analysis results output from the machine model that analyzes medical images of a subject, as shown in Figure 3, and information regarding a region of interest within the medical image that was focused on during the process of outputting the analysis results.

[0020] Figure 3 is a schematic diagram showing an example of a display image (display screen) 100 according to Embodiment 1. Specifically, Figure 3 is a display image in which the analysis results derived by a machine model that analyzes an image and information regarding the basis for deriving the analysis results are displayed on a single screen. Information regarding the basis for deriving the analysis results means at least one reason why the machine model derived such an analysis result. For example, information regarding the basis for deriving the analysis results may be information indicating which part of the image (medical image) was mainly used to derive the analysis result, or it may be information indicating which part was given what weight in the derivation. A display image is an image displayed on the screen (display screen) of a display device. The display image may be a browser image, that is, an image displayed on a website established on the internet that can be viewed using a browser. The type of display device is not limited. The screen may be, for example, the screen of a stationary personal computer or the screen of a mobile terminal.

[0021] Figure 3 specifically shows the analysis results derived by the machine model 601, which analyzes images containing bone, and information regarding the basis for deriving those analysis results, all displayed on a single screen (the screen displaying the image 100). The analysis results are displayed in region 101 and include bone density and the percentage of that bone density compared to the average bone density of young adults. Information regarding the basis for deriving those analysis results is displayed in region 102. Region 102 displays an image containing bone analyzed by the machine model 601, with the region of interest 103 added. In this embodiment, the region of interest is the region that served as the main basis for deriving the image analysis results. The "Future Prediction" button 105, the "Back" button 106, and the "Exit" button 107 will be described later.

[0022] As shown in Figure 3, by displaying the analysis results derived by the machine model 601 that analyzes the image, along with information on the basis for deriving those analysis results, on a single screen, the user can confirm at least one of the reasons why the machine model derived those results, along with the analysis results themselves.

[0023] By using the image generation device 3 or image display method S1 described above, the analysis results derived by the machine model that analyzes the image, and information regarding the basis for deriving those analysis results, can be displayed on a single screen. This allows the user to confirm, along with the analysis results, one of the reasons why the machine model derived those results, thereby improving confidence in the analysis results obtained using the machine model compared to displaying only the analysis results.

[0024] [Embodiment 2] Other embodiments of the present disclosure are described below. In this embodiment, an example of a displayed image will be described, taking as an example a case in which an analysis system (also called an image remote analysis system) 70A, which receives an image analysis request from a user, analyzes an X-ray image of bone and outputs bone density, relative comparison of bone densities, fracture probability (%), etc. The image remote analysis system 70A can display to the user, along with the analysis results, one of the reasons why the machine model derived such analysis results.

[0025] (Configuration of Image Remote Analysis System 70A) Figure 4 is a block diagram showing the configuration of the image remote analysis system 70A according to Embodiment 2 of the present disclosure. As shown in the figure, the image remote analysis system 70A comprises an analysis device 1A (image remote analysis device) for analyzing medical data (e.g., medical image data) and a user terminal 20, and the analysis device 1A and the user terminal 20 are connected to each other via the Internet.

[0026] The analysis device 1A comprises a communication unit 15, a control unit 16, and a storage unit 17. The communication unit 15 communicates with the user terminal 20 via the Internet. The storage unit 17 stores acquired information 172, which includes image data 301, encrypted patient information 302, and attribute information 303, and analysis result data 173, which includes processed image data 310 and derivation basis data 311. The storage unit 17 stores the image data 301, encrypted patient information 302, and attribute information 303 acquired by the acquisition unit 12, linking them together.

[0027] The control unit 16 comprises a communication control unit 11 (analysis result transmission unit), an acquisition unit 12, an analysis unit 13, a generation unit (image generation unit) 18, and a determination unit 19. The communication control unit 11 controls the communication unit 15. The acquisition unit 12 acquires data transmitted from the user terminal 20 via the communication control unit 11. The acquisition unit 12 also acquires analysis results (analysis data) derived by the analysis unit 13 (machine model 131) which analyzes medical images, and information regarding the basis for the derivation of said analysis results (derivation basis data). The analysis unit 13 analyzes image data 301. The control unit 16 may have the same configuration as the control unit 30 described in Embodiment 1.

[0028] The generation unit 18 generates information regarding the basis for the derivation of the analysis results analyzed by the analysis unit 13 (hereinafter referred to as "derivation basis data 311"). The derivation basis data 311 means at least one reason why the machine model derived such an analysis result. For example, the derivation basis data 311 may be information indicating which part of the image was mainly used to derive the analysis result. The derivation basis data 311 is linked to at least the encrypted patient information 302 and stored in the storage unit 17. Specific examples of the functions of the generation unit 18 will be described later. The generation unit 18 also generates a display image (display image data) in which the analysis results acquired by the acquisition unit 12 and the area of ​​interest information (information regarding the basis for the derivation of the analysis result) are displayed on one screen. The generation unit 18 records the generated display image data in the storage unit 17.

[0029] (Judgment section 19) The determination unit 19 determines whether the encrypted patient information 302 acquired by the acquisition unit 12 is actually encrypted patient identification information 320. If it is determined that the encrypted patient information 302 is not actually encrypted, the acquisition unit 12 may delete the acquired encrypted patient information 302. By providing such a determination unit 19, if the encrypted patient information 302 transmitted from the user terminal 20 is not encrypted patient identification information 320, the acquired image data 301 is deleted along with the encrypted patient information 302 and attribute information 303 without being analyzed. Therefore, the acquired information 172, including the image data 301, is not stored inside the analysis device 1A. The determination of whether or not something is encrypted may be made, for example, from the file extension of the encrypted data.

[0030] If it is determined that the encrypted patient information 302 is not actually encrypted, the control unit 16 may send a message to the user terminal 20 via the communication control unit 11 requesting that the patient identification information 320 be encrypted and sent again.

[0031] (Browser image 200) Figure 5 is a schematic diagram showing an example of a browser image 200 that is displayed on a web page when a user accesses the analysis device 1A via the internet, for inputting image data 301, etc.

[0032] As shown in the diagram, the browser image 200 displays the text "<Bone Density Analysis>" at the top, indicating the subject of the image analysis. The text displayed at the top is not limited to this; for example, it could be any indication of the analysis content, such as "<Bone Density Estimation>" or "<Bone Density Future Prediction>". The upper left of the browser image 200 may contain status information indicating the current phase or content of the analysis. For example, the upper left of the browser image 200 could, but is not limited to, "<Reception>", "<Analysis Results>", "<Future Prediction>", or "<Benign / Malignant Determination>".

[0033] Furthermore, in the upper left corner of browser image 200, the text "<Reception>" is displayed, for example, to indicate that it is a screen for accepting input. Below that, the text "Please place the image to be analyzed here" prompts the user to input the image data to be analyzed, and a frame 205 is displayed indicating the area for pasting (dragging and dropping) the image data. Further below that, the text "Please enter the attribute information here" prompts the user to input patient identification information 320 and attribute information 303, and an input box 206 is displayed. In the lower right corner of browser image 200, a "Submit" button 204 prompts the user to submit, and in the upper right corner, a "Back" button 207 is displayed to return to the initial screen of the website.

[0034] Figure 6 is a schematic diagram showing X-ray image data 300 for which bone density analysis is requested, as an example of image data 301. The X-ray image data 300 may be simple X-ray images such as lumbar X-ray images or chest X-ray images, or it may be an X-ray image taken with a DXA (Dual energy X-ray Absorptiometry) device. In a DXA device that measures bone density using the DXA method, when measuring bone density of the lumbar spine, X-rays are irradiated from the front of the subject's lumbar spine. Also, in a DXA device, when measuring bone density of the proximal femur, X-rays are irradiated from the front of the subject's proximal femur. The ultrasound method is a method of measuring bone density by applying ultrasound to bones such as the heel and tibia. Here, "front of the lumbar spine" and "front of the proximal femur" refer to the direction that correctly faces the imaging site such as the lumbar spine and proximal femur, and may be the ventral side of the subject's body or the posterior side of the subject's body. Microdensitometry (MD) involves irradiating the hand with X-rays. Ultrasound is a method of measuring bone density by applying ultrasound to bones such as the lumbar spine, femur, heel, or tibia. Furthermore, the image data does not have to be an X-ray image; any image containing bone information is acceptable. For example, it can be estimated from MRI (magnetic resonance imaging) images, CT (computed tomography) images, PET images, and ultrasound images.

[0035] In this embodiment, when a user accesses the analysis device 1A via the Internet, the communication control unit 11 may display a browser image 200 on a web page and send an encryption application via the communication unit 15. When the user enters the image data 301, patient identification information 320, and attribute information 303 to be analyzed and clicks the "Send" button 204, the patient identification information 320 is encrypted, and the encrypted patient information 302, along with the image data 301 and attribute information 303, is sent to the analysis device 1A. Therefore, the user does not need to perform the encryption process for the patient identification information 320. The encryption process may be performed after the user clicks the "Send" button 204, rather than when the user clicks it.

[0036] The encryption method can be any known method and is not limited to it. For example, it may be an encryption method that combines a public key and a private key. The encryption may be such that even the operator of the remote image analysis system 70A cannot decrypt it. This reduces the risk that the combination of image data 301 and patient identification information 320 may be leaked to the operator of the remote image analysis system 70A, even if the user requests image analysis from the operator.

[0037] The acquisition unit 12 acquires the X-ray image data 300, encrypted patient information 302, and attribute information 303 transmitted from the user terminal 20, and transmits them to the analysis unit 13.

[0038] The analysis unit 13 inputs the X-ray image data 300 transmitted from the acquisition unit 12 to the machine model 131, processes the output data from the machine model 131 as needed, and generates analysis result data 173. The generated analysis result data 173 is linked to at least encrypted patient information 302 and attribute information 303 and stored in the storage unit 17.

[0039] The communication control unit 11 acquires analysis result data 173 linked to encrypted patient information 302 and attribute information 303 from the storage unit 17 and transmits it to the user terminal 20 via the communication unit 15.

[0040] When the user terminal 20 receives the analysis result data 173, the encrypted patient information 302 is automatically decrypted and patient identification information 320 is generated. The encrypted patient information 302 may be decrypted when the analysis result data 173 is transmitted, or the user may decrypt it themselves.

[0041] As a result, the user terminal 20 can display the analysis result data 173 on its screen along with patient identification information 320, which includes the patient's name or identification number.

[0042] Figure 7 is a schematic diagram showing an example of a browser image 400 displayed on the screen of a user terminal 20 that has received the analysis result data 173. Browser image 400 is an example of display image data generated by the generation unit 18. At the top of browser image 400, the words "<Bone Density Analysis>" are displayed to indicate the content of the image analysis. At the top left of browser image 400, the words "<Analysis Results>" are displayed to indicate that this is a screen displaying the analysis results. Below that, patient information may be displayed. Below that, it may say, "Your bone density is □ / cm 2 The text "is" is displayed. Box 401 displays the estimated bone density value. Further below, the text "It is □% compared to young people" may be displayed. Box 402 displays the ratio to the Young Adult Mean (YAM) bone density. Further below, the text "Your femoral fracture probability is □%" may be displayed. Box 406 displays the probability of fracture as a percentage. Further below, the text "Judgment" and in box 403, text such as "Bone loss" may be displayed. For example, if the ratio to YAM is below 80%, it is judged as "Bone loss," and if it is below 70%, it is judged as a possibility of "Osteoporosis." Browser image 400 may have an "Exit" button 404 and a "Back" button 405. The "Back" button may, for example, return to the previous screen, return to the home screen, or return to a specific screen.

[0043] (Generation part 18) Next, the details of the generation unit 18 will be described. In this embodiment, the generation unit 18 generates the derivation basis data 311 of the analysis results analyzed by the analysis unit 13. The communication control unit 11 may transmit the derivation basis data 311 generated by the generation unit 18 to the user terminal 20.

[0044] The analysis result data 173 is based on the output from the machine model 131, but the output from the machine model 131 does not include the analysis process. Therefore, it is generally not possible to judge the reliability of the output by looking only at the output from the machine model 131. By sending the analysis result data 173, including the derivation basis data 311, to the user terminal 20, it is possible to help improve the user's confidence in the analysis results.

[0045] For example, the derived basis data 311 may be processed image data 310 (basis image data) to which new information has been added to the image data 301 input to the machine model 131. For example, the new information may be information indicating the region in the image data 301 that was the main basis for deriving the analysis result. The processed image may be an image to which information indicating the main basis region has been added to the input image data. Information indicating a region is information that shows the extent of the region, such as coloring or a frame. In image analysis, a certain region of the image is often the main basis for estimation, so such information indicating a region allows the user to confirm the region that was the basis for estimation. In this case, the display image shown on the user terminal 20 includes a processed image to which the area of ​​interest information has been added to the medical image input to the machine model 131. The generation unit 18 may generate a display image in which the medical image (image data 301) and the processed image are displayed side by side. Such a display image makes it easy for the user to compare the original analyzed image and the processed image. Alternatively, the control unit 16 may switch between displaying the medical image and the processed image by user operation. For example, the control unit 16 may alternately display medical images and processed images when the user clicks a switching button.

[0046] Furthermore, the processed image data 310 does not need to contain all the information of the user-input image data 301. For example, the processed image data 310 may be a cropped version of the input image data 301, or it may be an image with a lower resolution than the input image data 301. This reduces the amount of data transmitted and received.

[0047] Figure 8 is a schematic diagram showing an example of a browser image 500 that transmits the derivation basis data 311 in addition to the analysis result data 173 to the user terminal 20. The browser image 500 is an example of display image data generated by the generation unit 18. The browser image 500 has the derivation basis data 311 (processed image data 502 including region 503) added to the analysis result data 173 shown in Figure 7.

[0048] Specifically, the browser image 500 displays the analysis result 501 along with the processed image data 502, which is the input image data (X-ray image data 300 in Figure 6) with a rectangular region 503 added. As shown in Figure 8, in this embodiment, the analysis result 501 derived by the machine model 131 that analyzes the image data, and the processed image data 502 including the derivation basis data 311 are displayed on a single screen. The browser image 500 may also display a "Back" button 506, an "Exit" button 507, and a "Future Prediction" button 505. The role of the "Future Prediction" button 505 will be described later.

[0049] Figure 9 is a magnified schematic diagram of the image of region 503 in Figure 8. Region 503 includes the four lumbar vertebrae shown from L1 to L4. This indicates that lumbar vertebrae L1-L4 are the region on which the analysis results were based. In fact, it is known that the bone density of lumbar vertebrae L1-L4 is related to the average bone density of the whole body. That is, the enclosed region 503 indicates that the analysis results of the mechanical model 131 were derived based on the bone density of these lumbar vertebrae L1-L4.

[0050] The area of ​​focus on which the analysis results are based may include a segmented area of ​​the medical image. Segmentation is the process of dividing an image into several regions. Segmentation is performed to reduce the processing load of the machine model 131. That is, the machine model 131 may analyze only the segmented area. The segmented area can be set to any range. The segmented area may be, for example, a rectangle, a square, or a circle. If the segmented area is a square, the processing load of the machine model 131 can be reduced. When analyzing an X-ray image of the lumbar spine, for example, the area including lumbar vertebrae L1 to L4 is segmented. Therefore, the size of the segmented area may vary depending on the size of lumbar vertebrae L1 to L4 in the image. The segmented area may be set to be slightly larger than lumbar vertebrae L1 to L4 in the direction of the arrangement of lumbar vertebrae L1 to L4, for example, or it may be set to overlap with the edges of lumbar vertebrae L1 to L4. The segmentation region may always have predetermined dimensions, or its dimensions may be set according to the medical image. For example, the segmentation region may identify the positions of lumbar vertebrae L1-L4, set the length of the lumbar vertebrae L1-L4 in the alignment direction, and then set the length of the lumbar vertebrae L1-L4 in the vertical direction. Segmentation may be performed by a machine model 131. The machine model 131 may have learned images with annotations of the analysis region in order to perform segmentation.

[0051] Furthermore, the generation unit 18 may generate a heatmap of the region that formed the basis of the analysis results. In this case, for example, the outer edge of the heatmap would point to the segmentation region. A heatmap is a method of representing the magnitude of bone density with the intensity of any color. For example, the generation unit 18 may generate a heatmap that shows the degree of focus. It may also generate a heatmap that shows the numerical value of bone density. Furthermore, the generation unit 18 may generate a heatmap that shows the possibility (probability) of fracture. The processed image may be an image in which a heatmap showing the region of focus information is superimposed on a medical image. The image used for the heatmap may be a still image or a video. By showing it as a video, for example, by fading various heatmaps in order, it becomes easy to visually recognize the relationships between each heatmap. Furthermore, if the analysis result is a heatmap of bone density that includes areas other than the segmentation region, a portion of the segmentation region may be enclosed in a frame.

[0052] The generation unit 18 may obtain information about the region that formed the basis of the analysis results from the analysis unit 13. Specifically, the generation unit 18 may obtain the region that formed the basis of the analysis results from the analysis unit 13 and generate information indicating that region (such as a border surrounding region 503). The generation unit 18 may obtain data indicating the degree of focus, bone density data, or information indicating the possibility of fracture within the region from the analysis unit 13 and generate a heat map. The generation unit 18 may generate any multiple of the following: a heat map showing the degree of focus data, a heat map showing bone density data, and a heat map showing the possibility of fracture. When displaying the heat map showing the degree of focus data, the heat map showing bone density data, and the heat map showing the possibility of fracture overlaid, the generation unit 18 may make the colors of the heat map showing the degree of focus data, the heat map showing bone density data, and the heat map showing the possibility of fracture different.

[0053] The machine model 131 analyzes, for example, X-ray image data 300 using a neural network model (NNM). In the NNM, the image is divided into small regions, each is quantified, and multiple regions are pooled to integrate them into larger regions, which are then quantified again. This process is repeated. Therefore, the machine model 131 may extract regions that have numerical values ​​that influence the processing results (for example, relatively large numerical values) as the basis regions.

[0054] In the example shown in Figure 8, a processed image data 502 was described in which the region 503 that formed the basis of the analysis result was superimposed on the X-ray image data 300 that was to be analyzed. However, the processed image data is not limited to this. For example, the analysis device 1 may transmit positional information (such as coordinates) of the region that formed the basis of the analysis result in the image to be analyzed to the user terminal, and the user terminal may display the processed image data with the region superimposed on the image to be analyzed.

[0055] A screen like the one shown in Figure 8 can be used when a doctor explains the analysis results to a patient. In other words, it can not only explain the analysis results to the patient, but also explain which areas of the image data were used as the basis for these results. This not only improves the doctor's confidence in the analysis results, but also makes it easier for the patient to accept the results.

[0056] In the example shown in Figure 8, the analysis results 501 and the processed image data 502, which includes the derivation basis data 311, are displayed on a single screen. However, "a single screen" does not necessarily mean that the two are displayed simultaneously on the screen. For example, the two may be displayed by scrolling the screen up and down or left and right. In other words, the range displayed by scrolling the screen up and down or left and right is referred to as "a single screen."

[0057] Figure 10 is a schematic diagram showing an example of a browser image 700 that displays the location of a bone fracture in a patient at the present time, estimated from the image data analyzed by the analysis unit 13, the probability of that location fracturing, and the probability of that location fracturing in the patient three years from now. Note that the future time is not limited to three years from now, but can be any time (for example, X years from now). Browser image 700 is an example of display image data generated by the generation unit 18. This can be displayed by clicking the "Future Prediction" button 505 in the lower right corner of the browser image 500 shown in Figure 8.

[0058] Browser image 700 displays the text "<Bone Density Analysis>" at the top, indicating the subject of the image analysis. In the upper left of browser image 700, the text "<Future Prediction>" is displayed, indicating that this is a screen showing future predictions. Below that, patient information may be displayed. Below that, the text "Your femoral fracture probability is □%" is displayed. Box 701 displays the currently estimated femoral fracture probability. Further below, the text "Your femoral fracture probability in 3 years is □%" may be displayed. Box 702 displays the predicted fracture probability in 3 years. In addition, information regarding the basis for estimation or prediction may be displayed in conjunction with this image.

[0059] Figure 11 shows an example of a browser image 800 in which the derivation basis data has been added to the display image shown in Figure 10. Browser image 800 is an example of display image data generated by the generation unit 18. As shown in Figure 11, the browser image 800 displays an image 802 showing the derivation basis data in addition to the region 801 that shows the femoral fracture probability at the present time and 3 years from now, which was derived by the machine model 131.

[0060] Such browser images 800 can be used by physicians to explain the current and future risk of fractures to patients. Displaying information on the underlying rationale or predictive basis alongside the image can increase its persuasiveness to the patient.

[0061] (Machine Model 131) Next, the machine model 131 will be described. The machine model 131 is a model that estimates the bone condition of a subject. The input image data is an image containing bone, and the analysis result outputs an estimated result regarding the bone condition. For example, the machine model 131 is a trained model that has been trained to output estimated or calculated results regarding the bone condition, such as bone density, relative comparison of bone densities, and the probability of fracture, from X-ray images of bone. The bone density may be the calculated bone density of the bone region included in the image data, or it may be the average bone density of the whole body estimated from the image data. A known method can be used to calculate the bone density from the image. The relative comparison of bone densities is the ratio of the estimated bone density to YAM. The probability of fracture is the possibility of a fracture in a specific area (e.g., the femoral neck). The estimated result regarding the bone condition may be the estimated result at the time the image was taken, or it may be a prediction at a predetermined period of time after that time.

[0062] The machine model 131 may output at least one of the following as estimation results: bone density estimated at the time image data is acquired, bone density predicted after a predetermined period has elapsed from the time image data is acquired, fracture site and its probability estimated at the time image data is acquired, and fracture site and its probability predicted after a predetermined period has elapsed from the time image data is acquired.

[0063] (Learning method for machine model 131) Next, the learning method for the machine model 131 will be described. For example, learning to estimate bone density may be performed using X-ray images of bones with identified bone density as training data. Learning to estimate the likelihood of fracture may be performed using X-ray images of bones and data on whether the patient subsequently suffered a fracture within a predetermined period as training data. Relative comparison of bone density does not need to be learned; it can be obtained by dividing the estimated bone density by YAM. Furthermore, learning for future prediction may be performed using X-ray images of bones with identified bone density and data on what the patient's bone density was after a predetermined period, or whether or not a fracture occurred, as training data. By including lifestyle habits such as exercise, diet, smoking, and alcohol consumption of the patients used as training data, it is possible to construct a machine model 131 that can estimate or predict with even greater accuracy.

[0064] In Embodiment 2, a machine model 131 trained to estimate bone conditions such as bone density was used as an example. In this case, the input image data is an X-ray image of bone, and the output is an estimation result regarding the bone condition. However, the machine model 131 is not limited to this type of model. For example, the machine model 131 may be a cell pathology analysis model. In this case, the input medical image may be a microscopic image of the subject's cells, and the analysis result may be information regarding pathological mutations in those cells.

[0065] Figure 12 is a schematic diagram showing an example of a browser image 900, which is a display image with the cell pathology analysis results and the data for which they were derived added, when the machine model 131 is a cell pathology analysis model. Browser image 900 is an example of display image data generated by the generation unit 18. In the browser image 900 shown in Figure 12, the words "<Cell Pathology Analysis>" are displayed at the top to indicate the content of the image analysis. In the upper left of browser image 900, the words "<Judgment Result>" are displayed to indicate that this is a screen that displays the result of determining whether the cells were benign or malignant. Below that, patient information may be displayed. In the area 901 below that, the words "The probability of malignancy is □%" are displayed. The box displays a numerical value indicating the probability that the cells are malignant. Furthermore, to the right of area 901, image 902 is displayed with area 903 added as the data for which the analysis results were derived, which was the main basis for deriving the judgment result from the analyzed image. Area 903 is the area containing the cells that the machine model 131 determined to be malignant. Additionally, the browser image 900 may also display an "Exit" button 904 and a "Back" button 905.

[0066] Furthermore, a single screen may refer to a screen that is displayed by scrolling. Figure 13 is a schematic diagram showing an example of displaying images on a single screen by scrolling. The solid line region 1301 in Figure 13 is the display area of ​​the user terminal 20 (display device). The dotted line region 1302 is a single screen. The upper part of region 1302 displays the bone density analysis result 501 data, as explained in Figure 8. The lower part of region 1302 displays the processed image data 502, which is the basis derivation data, as explained in Figure 8. When region 1302 is first displayed, the bone density analysis result 501 data is displayed in region 1301, as shown in the upper part of Figure 13. However, when the user scrolls region 1302 upwards, the processed image data 502 appears, as shown in the upper part of Figure 13. Even with this configuration, the user can display the analysis results and the basis data for their derivation simply by scrolling the screen.

[0067] According to the configuration of the image remote analysis system 70A in the above embodiment 2, the derived basis data 311 can be provided to the user together with the analysis result data 173. This has the effect of improving the user's confidence in the analysis results. In addition, it has the effect of making it easier for patients to accept the analysis results when explaining them to them.

[0068] Next, the flow of the image remote analysis method S2 executed by the control unit 16 according to Embodiment 2 will be described. The image remote analysis method S2 includes steps S21 to S24 (not shown).

[0069] When a user accesses the analysis device 1A, the communication control unit 11 displays a browser image 200 on a web page via the communication unit 15, which is an input screen for inputting acquired information 172, including image data 301 to be analyzed, patient identification information 320, and attribute information 303 (S21).

[0070] The acquisition unit 12 acquires encrypted patient information 302, which is encrypted image data 301, attribute information 303, and patient identification information 320 contained in the acquired information 172 that is input into the browser image 200 and transmitted from the user terminal 20 (S22).

[0071] The analysis unit 13 analyzes the image data 301 sent from the acquisition unit 12 using the machine model 131 (S23).

[0072] The communication control unit 11 transmits the analysis result data 173 analyzed by the analysis unit 13, the derived basis data 311, and the encrypted patient information 302 to the user terminal 20 (S24).

[0073] According to the image remote analysis method S2 described above, the derived basis data 311 can be provided to the user together with the analysis result data 173. This has the effect of improving the user's confidence in the analysis results. In addition, it has the effect of making it easier for patients to accept the analysis results when explaining them to them.

[0074] [Examples of implementation using software] The functions of the image remote analysis system 70A (hereinafter referred to as "the system") are programs that cause a computer to function as the system, and these functions can be realized by programs that cause a computer to function as each part of the system.

[0075] In this case, the system includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the program. By executing the program using this control device and storage device, the functions described in each of the embodiments are realized.

[0076] The above program may be recorded on one or more computer-readable recording media, not temporary ones. These recording media may or may not be provided by the above device. In the latter case, the program may be supplied to the above device via any wired or wireless transmission medium.

[0077] Furthermore, some or all of the functions of the above-mentioned parts can also be realized by logic circuits. For example, an integrated circuit in which logic circuits functioning as the above-mentioned parts are formed is also included in the scope of this disclosure. In addition, it is also possible to realize the functions of the above-mentioned parts by, for example, a quantum computer.

[0078] The inventions described in this disclosure have been explained above based on the drawings and embodiments. However, the inventions described in this disclosure are not limited to the embodiments described above. That is, the inventions described in this disclosure can be modified in various ways within the scope shown in this disclosure, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the inventions described in this disclosure. In other words, it should be noted that it is easy for those skilled in the art to make various modifications or alterations based on this disclosure. Furthermore, it should be noted that these modifications or alterations are included in the scope of this disclosure.

[0079] This disclosure describes, but is not limited to, the analysis of medical images and the modification of medical images based on focus region information related to the focus region identified during the process of outputting the analysis results. For example, analysis may be performed using a pre-trained machine model with only numerical data or the information shown below as input information, without including medical images. In this case, the focus element information identified during the process of outputting the analysis results may be displayed on the displayed image along with the analysis results. Furthermore, the focus element information may be a part of the input information that has been highlighted. For example, if age and gender information are focus elements, the age and gender items or numerical values ​​may be highlighted in a conspicuous color.

[0080] The input information may include age, sex, weight, height, presence or absence of fractures, location of fractures, fracture history, family history of fractures (e.g., parents), underlying medical conditions (e.g., food and / or drug allergies, diseases related to and unrelated to the onset of osteoporosis, etc.), smoking history, drinking habits (e.g., frequency and amount of alcohol consumed, etc.), occupational history, exercise history, medical history (e.g., history of bone disease), menstruation (e.g., cycle and presence or absence, etc.), menopause (e.g., possibility and presence or absence, etc.), artificial joints (e.g., type, presence or absence, and timing of replacement surgery for spinal implants or knee joints, etc.), blood test results, urine test results, medications being taken, and gene sequences.

[0081] (summary) (Aspect 1) An image generation apparatus according to Embodiment 1 of the present disclosure comprises: an acquisition unit that acquires analysis results output from a machine model that analyzes medical images of a subject, and focus region information relating to a region within the medical image that was focused on during the process of outputting the analysis results; and an image generation unit that generates a display image by modifying the medical image based on the focus region information.

[0082] (Aspect 2) In the image generation apparatus according to aspect 2 of this disclosure, in aspect 1 above, the displayed image includes a processed image to which the area of ​​interest information has been added to the medical image input to the machine model.

[0083] (Aspect 3) In the image generation apparatus according to embodiment 3 of the present disclosure, in embodiment 1 or 2 above, the region of interest includes the region in which the medical image is segmented.

[0084] (Aspect 4) In the image generation apparatus according to embodiment 4 of this disclosure, the area of ​​interest information is shown as a heat map in any one of embodiments 1 to 3 above.

[0085] (Aspect 5) In the image generation apparatus according to Embodiment 5 of this disclosure, in Embodiment 2 above, the processed image is an image on which the area of ​​interest information is superimposed.

[0086] (Aspect 6) In the image generation apparatus according to embodiment 6 of the present disclosure, in embodiment 2 or 5 above, the image generation unit generates the display image in which the medical image and the processed image are placed side by side.

[0087] (Aspect 7) In the image generation apparatus according to embodiment 7 of this disclosure, in any one of embodiments 2, 5, or 6 above, the display image and the processed image can be swapped by operation.

[0088] (Pattern 8) The image generating apparatus according to aspect 8 of the present disclosure is, in any one of aspects 1 to 7 above, wherein the machine model is a model for estimating the bone condition of the subject, the medical image is an image showing the bones of the subject, and the analysis result includes an estimation result regarding the bone condition of the subject.

[0089] (Aspect 9) In the image generation apparatus according to embodiment 9 of the present disclosure, in any one of embodiments 1 to 8, the displayed image is shown as a heatmap in which the analysis result and the area of ​​interest information are each in different embodiments.

[0090] (Aspect 10) In the image generation apparatus according to aspect 10 of the present disclosure, in aspect 8, the estimation result is at least one of the following: the estimated location of the fracture of the subject, and the probability that the subject will suffer a fracture.

[0091] (Aspect 11) The image generating apparatus according to embodiment 11 of the present disclosure is, in any one of embodiments 1 to 7 above, wherein the machine model is a cell pathology analysis model, the medical image is a microscopic image of the subject's cells, and the analysis result includes information on pathological mutations in the subject's cells.

[0092] (Aspect 12) The image generating apparatus according to aspect 12 of this disclosure is an image display that is displayed on a single screen, in any one of aspects 1 to 11 described above.

[0093] (Aspect 13) An image generation method according to aspect 13 of this disclosure includes an acquisition step of acquiring analysis results output from a machine model that analyzes medical images of a subject, and focus region information relating to a region within the medical image that was focused on during the process of outputting the analysis results, and an image generation step of generating a display image by modifying the medical image based on the focus region information.

[0094] (Aspect 14) The program according to aspect 14 of this disclosure is an image generation program for causing a computer to function as an image generation device as described in claim 1, and is an image generation program for causing a computer to function as the acquisition unit and the image generation unit.

[0095] (Aspect 15) The recording medium according to aspect 15 of this disclosure is a computer-readable non-temporary recording medium on which the image generation program described in claim 14 is recorded. [Explanation of symbols]

[0096] 70,70A...Image Remote Analysis System 1A...Analysis device 3. Image generation device 11,33...Communication Control Unit 12...Acquisition part 13...Analysis department 131... Mechanical Model 14. Analysis Result Transmission Section 15,50... Communications Department 16,30...Control Unit 17,40...Storage section 18...Generation section 19... Judgment section 20. User terminals 31. Image generation unit 32. Data acquisition unit (acquisition unit) 41. Analysis data 42. Derivation basis data 43...Displayed image data 60...Analysis equipment 601... Mechanical Model 70...Display device

Claims

1. An acquisition unit that acquires analysis results regarding the state of bones output based on a machine model from input information including an X-ray image showing the bones of a subject, and information on a region of interest within the X-ray image that the machine model used as the basis in the process of deriving the analysis results, The system includes an image generation unit that generates a display image by modifying the X-ray image based on the aforementioned area of ​​interest information, The image generation system wherein the displayed image is an image that includes the analysis results and the information of the area of ​​interest.

2. The image generation system according to claim 1, wherein the area of ​​interest includes at least a portion of the bones of the subject.

3. The image generation system according to claim 1, wherein the area of ​​interest includes at least a portion of the lumbar spine of the subject.

4. The image generation system according to claim 1, wherein the area of ​​interest is rectangular.

5. The image generation system according to claim 1, wherein the displayed image includes a processed image to which the area of ​​interest information has been added to the X-ray image input to the machine model.

6. The image generation system according to claim 1, wherein the information of the area of ​​interest is shown in a heat map.

7. The image generation system according to claim 5, wherein the processed image is an image on which the area of ​​interest information is superimposed.

8. The image generation system according to claim 5, wherein the image generation unit generates the display image in which the X-ray image and the processed image are arranged in parallel.

9. The image generation system according to claim 5, wherein the X-ray image and the processed image can be swapped by operation.

10. The analysis results include estimations regarding the bone condition of the subject, The image generation system according to claim 1, wherein the estimation result is at least one of the following: the estimated location of the fracture of the subject, and the probability that the subject will suffer a fracture.

11. The analysis results include estimations regarding the bone condition of the subject, The image generation system according to claim 1, wherein the estimation result includes the analysis result regarding the bone density of the subject.

12. The image generation system according to claim 1, wherein the region of interest includes at least a portion of a segmented region which is at least a part of the X-ray image.

13. The image generation system according to claim 12, wherein the machine model performs analysis of the segmented region.

14. The image generation system according to claim 13, wherein the machine model performs segmentation of the X-ray image.

15. An acquisition step of acquiring analysis results regarding the state of bones output based on a machine model from input information including an X-ray image showing the bones of a subject, and information on a region of interest within the X-ray image that the machine model used as the basis in the process of deriving the analysis results, The image generation step includes generating a display image by modifying the X-ray image based on the aforementioned area of ​​interest information, An image generation method wherein the displayed image is an image that includes the analysis results and the area of ​​interest information.

16. An image generation program for causing a computer to function as an image generation system according to claim 1, wherein the computer functions as the acquisition unit and the image generation unit.

17. A computer-readable non-temporary recording medium that stores the image generation program described in claim 16.

Images