Image generating device, image generating method, image generating program, and recording medium

Abstract

To improve a feeling of trust in an analysis result obtained by using a machine model.SOLUTION: An image creation device (3) comprises: an acquisition unit (32) that acquires an analysis result (41) output from a machine model (601) for analyzing a medical image obtained by picking up an image of a subject, and attention area information (42) related to an attention area that is an area in the medical image to which attention is paid in a process in which the analysis result is output; and an image creation unit (31) that creates a display image obtained by changing the medical image based on the attention area information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image generating apparatus, an image generating method, an image generating program, and a recording medium. [Background technology]

[0002] For example, Patent Document 1 discloses an X-ray age estimation model that uses X-ray image data of a subject and information on the subject's age at the time the X-ray image was taken, and shows the correspondence between features obtained from the X-ray image data and the subject's age. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-148729 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, various images have been analyzed using trained machine models. However, the basis of the analysis results obtained using the machine models is often unclear. The purpose of the present disclosure is to further improve the reliability of the analysis results obtained using the machine models. [Means for solving the problem]

[0005] An image generating device according to one aspect of the present disclosure includes an acquisition unit that acquires analysis results output from a machine model that analyzes a medical image of a subject and area of ​​interest information relating to an area of ​​interest within the medical image that is focused on during the process of outputting the analysis results, and an image generating unit that generates a display image by modifying the medical image based on the area of ​​interest information.

[0006] An image generation method according to one aspect of the present disclosure includes an acquisition step of acquiring analysis results output from a machine model that analyzes a medical image of a subject and area of ​​interest information relating to an area of ​​interest within the medical image that is focused on during the process of outputting the analysis results, and an image generation step of generating a display image in which the medical image is modified based on the area of ​​interest information.

[0007] The image generating device according to each aspect of the present disclosure may be realized by a computer. In this case, the control program of the image generating device that causes the computer to operate as each part (software element) of the image generating device to realize the image generating device, and the computer-readable recording medium on which it is recorded, also fall within the scope of the present disclosure. Effect of the Invention

[0008] According to one aspect of the present invention, it is possible to further improve the reliability of analysis results obtained using a machine model. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an image generating device according to a first embodiment of the present disclosure. [Diagram 2] 1 is a flowchart showing the flow of an image display method S1 according to the first embodiment. [Diagram 3] FIG. 2 is a schematic diagram showing an example of a display image according to the first embodiment. [Figure 4] FIG. 11 is a block diagram showing a configuration of an image remote analysis system according to a second embodiment of the present disclosure. [Diagram 5] FIG. 13 is a schematic diagram showing an example of a browser image for user input; [Figure 6] FIG. 2 is a schematic diagram showing an example of X-ray image data for which a bone density analysis is requested. [Figure 7] 13 is a schematic diagram showing an example of a browser image of an analysis result transmitted by an analysis result transmitting unit; FIG. [Figure 8]8 is a schematic diagram showing an example of a browser image in which derived basis data is added to the display image shown in FIG. 7; FIG. [Figure 9] FIG. 9 is an enlarged schematic diagram of the image of the area 503 shown in FIG. [Figure 10] FIG. 13 is a schematic diagram showing an example of a browser image displaying the patient's current and future risk of fracture. [Figure 11] 11 is a schematic diagram showing an example of a browser image in which derived basis data is added to the display image shown in FIG. 10; [Figure 12] FIG. 13 is a schematic diagram showing an example of a display image to which a cytopathological analysis result and basis data for its derivation have been added. [Figure 13] FIG. 13 is a schematic diagram showing an example in which a display image is displayed on one screen by scrolling. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] [Embodiment 1] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an image generating device 3 according to the first embodiment of the present disclosure. The image generating device 3 is a device that generates a display image in which a medical image is modified so as to include the basis of an analysis result based on information on a region of interest that is focused on in a process in which an analysis result is output from a machine model that analyzes a medical image. The image generating device 3 is capable of communicating information with an analysis device 60 and a display device 70 via a communication unit 50. The analysis device 60 includes a machine model 601. The machine model 601 is, for example, a trained machine model that analyzes at least one of an image and a numerical value. The display device 70 displays, for example, a result analyzed by the machine model 601.

[0011] 1, the image generating device 3 includes a control unit 30, a storage unit 40, and a communication unit 50. The control unit 30 includes an image generating unit 31, a data acquiring unit (acquiring unit) 32, and a communication control unit 33.

[0012] The data acquisition unit 32 acquires the analysis result (analysis data) output (derived) from the machine model 601 that analyzes the image, and information on the basis for deriving the analysis result (derivation basis data). The derivation basis data is, for example, area of ​​interest information on an area of ​​interest in a medical image that is focused on during the process of outputting the analysis result. The data acquisition unit 32 records the acquired analysis data 41 and derivation basis data 42 in the storage unit 40.

[0013] The image generating unit 31 generates a display image (display image data) by changing the medical image based on the region of interest information. The image generating unit 31 records the generated display image data 43 in the storage unit 40.

[0014] The communication control unit 33 transmits the display image data 43 recorded in the storage unit 40 to the display device 70. In this case, the communication control unit 33 may transmit the display image data 43 to the display device 70. Alternatively, the communication control unit 33 may transmit the image data used for the analysis first, and then transmit the analysis data 41. This can increase the transmission speed.

[0015] The control unit 30 controls the entire image generating device 3. The control unit 30 includes at least one processor and at least one memory. The processor can be configured using a general-purpose processor such as at least one MPU (Micro Processing Unit) or CPU (Central Processing Unit). The memory may include multiple types of memory such as ROM (Read Only Memory) and RAM (Random Access Memory). As an example, the processor realizes the function of the image generating device 3 by expanding various control programs recorded in the ROM of the memory into the RAM and executing them. The processor may also include a processor configured with an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a PLD (Programmable Logic Device), or the like.

[0016] 2 is a flowchart showing, as an example, the flow of an image display method S1 using the image generating device 3. As shown in the figure, the image display method S1 includes steps S11 to S13.

[0017] The data acquisition unit 32 acquires the analysis result derived by the machine model 601 that analyzes the image and information on the basis for deriving the analysis result (S11). Specifically, for example, the data acquisition unit 32 acquires the analysis result output from the machine model that analyzes the medical image of the subject, and area of ​​interest information on an area in the medical image that is focused on in the process of outputting the analysis result. The area in the medical image may be, for example, an area inside the outer periphery of the medical image. The area of ​​interest information may be, for example, entirely located in the area in the medical image, or may overlap only partially. Alternatively, if the area of ​​interest information is not in the medical image (or, for example, the entire medical image), it may not overlap the area in the medical image. In this case, for example, a message indicating that the area of ​​interest does not exist (or, for example, that the entire medical image is the area of ​​interest) may be displayed outside the area in the medical image.

[0018] The image generating unit 31 generates a display image that displays the acquired analysis result and information on the basis for deriving the analysis result on one screen (S12). Specifically, the image generating unit 31 generates a display image in which the medical image is modified based on the region of interest information.

[0019] The communication control unit 33 transmits the display image generated by the image generating unit 31 to the display device 70 (step S13). The above processing method makes it possible to display on one screen the analysis results derived by the machine model that analyzes the image and information on the basis for deriving the analysis results. As an example, this method generates a display image 100 as shown in FIG. 3, in which the analysis results output from the machine model that analyzes the medical image of the subject and the area of ​​interest information on the area of ​​interest that is an area in the medical image and that is focused on during the process of outputting the analysis results are displayed on one screen, and the display device 70 displays the display image 100.

[0020] FIG. 3 is a schematic diagram showing an example of a display image (display screen) 100 according to the first embodiment. Specifically, FIG. 3 is a display image in which an analysis result derived by a machine model that analyzes an image and information on the basis of which the analysis result is derived are displayed on one screen. The information on the basis of which the analysis result is derived means at least one of the reasons why the machine model derived such an analysis result. For example, the information on the basis of which the analysis result is derived may be information indicating which part of the image (medical image) was mainly used to derive the analysis result, or information indicating which part was weighted and how much weight was given to the part. The display image is an image displayed on the screen (display screen) of a display device. The display image may be a browser image, that is, an image displayed on a website opened on the Internet that can be viewed using a browser. The type of display device is not limited. The screen may be, for example, a screen of a stationary personal computer or a screen of a mobile terminal.

[0021] Specifically, in FIG. 3, the analysis result derived by the machine model 601 that analyzes the image including bones and information on the basis for deriving the analysis result are displayed on one display screen (the screen displaying the display image 100). The analysis result is displayed in an area 101, and includes bone density and a ratio of the bone density compared to the average bone density of a young adult. Information on the basis for deriving the analysis result is displayed in an area 102. What is displayed in the area 102 is an image including bones analyzed by the machine model 601, to which a region of interest 103 has been added. In this embodiment, the region of interest is a region that is the main basis for deriving the analysis result of the image. The "future prediction" button 105, the "back" button 106 and the "end" button 107 will be described later.

[0022] As shown in FIG. 3, by displaying the analysis results derived by the machine model 601 analyzing the image and information regarding the basis for deriving the analysis results on a single screen, the user can confirm at least one of the reasons why the machine model derived such analysis results, along with the analysis results.

[0023] By using the image generating device 3 or image display method S1 described above, it is possible to display on one screen the analysis result derived by the machine model that analyzes the image and information on the basis for deriving the analysis result. This allows the user to confirm one of the grounds for why the machine model derived the analysis result together with the analysis result, thereby improving the reliability of the analysis result obtained using the machine model compared to when only the analysis result is displayed.

[0024] [Embodiment 2] Other embodiments of the present disclosure will be described below. In this embodiment, an example of a display image etc. will be described using an example in which an analysis system (also called an image remote analysis system) 70A, which receives an image analysis request from a user, analyzes an X-ray image of a bone and outputs bone density, a relative comparison of bone density, a possibility of fracture (%), etc. The image remote analysis system 70A can display to the user, together with the analysis result, one of the reasons why the mechanical model derived such an analysis result.

[0025] (Configuration of image remote analysis system 70A) 4 is a block diagram showing a configuration of an image remote analysis system 70A according to the second embodiment of the present disclosure. As shown in the figure, the image remote analysis system 70A includes an analysis device 1A (image remote analysis device) that analyzes medical data (e.g., medical image data) and a user terminal 20, and the analysis device 1A and the user terminal 20 are connected to each other so as to be able to communicate with each other via the Internet.

[0026] The analysis device 1A includes a communication unit 15, a control unit 16, and a storage unit 17. The communication unit 15 communicates with a user terminal 20 via the Internet. The storage unit 17 stores acquired information 172 including image data 301, encrypted patient information 302, and attribute information 303, and analysis result data 173 including processed image data 310 and derivation basis data 311. The storage unit 17 stores the image data 301, encrypted patient information 302, and attribute information 303 acquired by the acquisition unit 12 in a linked manner.

[0027] The control unit 16 includes a communication control unit 11 (analysis result transmission unit), an acquisition unit 12, an analysis unit 13, a generation unit (image generation unit) 18, and a judgment unit 19. The communication control unit 11 controls the communication unit 15. The acquisition unit 12 acquires data transmitted from the user terminal 20 via the communication control unit 11. The acquisition unit 12 also acquires analysis results (analysis data) derived by the analysis unit 13 (machine model 131) that analyzes the medical image, and information on the basis for derivation of the analysis results (derivation basis data). The analysis unit 13 analyzes the image data 301. The control unit 16 may have a configuration similar to that of the control unit 30 described in the first embodiment.

[0028] The generating unit 18 generates information on the basis of derivation of the analysis result analyzed by the analyzing unit 13 (hereinafter referred to as "derivation basis data 311"). The derivation basis data 311 means at least one of the basis of why the machine model derived such an analysis result. For example, the derivation basis data 311 may be information indicating which part of the image was mainly used to derive the analysis result. The derivation basis data 311 is linked to at least the encrypted patient information 302 and stored in the storage unit 17. A specific example of the function of the generating unit 18 will be described later. In addition, the generating unit 18 generates a display image (display image data) in which the analysis result acquired by the acquiring unit 12 and the area of ​​interest information (information on the basis of derivation of the analysis result) are displayed on one screen. The generating unit 18 records the generated display image data in the storage unit 17.

[0029] (Judgment section 19) The determination unit 19 determines whether the encrypted patient information 302 acquired by the acquisition unit 12 is the patient identification information 320 that has actually been encrypted. When it is determined that the encrypted patient information 302 is not the encrypted patient information, the acquisition unit 12 may delete the acquired encrypted patient information 302. By providing such a determination unit 19, if the encrypted patient information 302 transmitted from the user terminal 20 is not the encrypted patient identification information 320, the acquired image data 301 is not analyzed, and is deleted together with the encrypted patient information 302 and the attribute information 303. Therefore, the acquired information 172 including the image data 301 is not stored inside the analysis device 1A. The determination of whether the information is encrypted may be made, for example, from the extension of the encrypted data.

[0030] If it is determined that the encrypted patient information 302 has not actually been encrypted, the control unit 16 may send a message to the user terminal 20 via the communication control unit 11 requesting that the patient identification information 320 be re-encrypted and sent.

[0031] (Browser image 200) FIG. 5 is a schematic diagram showing an example of a browser image 200 displayed on a web page for inputting image data 301 and the like when a user accesses the analysis device 1A via the Internet.

[0032] As shown in the figure, the words "Bone Density Analysis" indicating the subject of image analysis are displayed at the top of the browser image 200. The words displayed at the top are not limited to this, and may be, for example, a display that indicates the content of the analysis, i.e., "Bone Density Estimation" and "Future Bone Density Prediction". The upper left corner of the browser image 200 may display status information that indicates the phase of the analysis in which the current screen is displayed or the content of the analysis. For example, the upper left corner of the browser image 200 may display, but is not limited to, "Reception", "Analysis Results", "Future Prediction", "Benign or Malignant Determination", etc.

[0033] Also, in the upper left corner of the browser image 200, for example, the word <RECEPTION> is displayed, indicating that this is a screen for accepting input. Below that, the words "Place the image to be analyzed here" are displayed, prompting the user to input image data to be analyzed, along with a frame 205 indicating an area for pasting (drag-and-dropping) the image data. Further below that, the words "Enter attribute information here" are displayed, prompting the user to enter patient identification information 320 and attribute information 303, etc., along with an input box 206. In the lower right corner of the browser image 200, a "SEND" button 204 is displayed, prompting the user to send the data, and in the upper right corner, a "BACK" button 207 is displayed, returning the user to the initial screen of the website.

[0034] FIG. 6 is a schematic diagram showing X-ray image data 300 for requesting bone density analysis as an example of image data 301. The X-ray image data 300 may be a simple X-ray image such as a lumbar X-ray image or a chest X-ray image, or may be an X-ray image taken by a DXA (Dual energy X-ray Absorptiometry) device or the like. In a DXA device for measuring bone density using the DXA method, when the bone density of the lumbar vertebrae is measured, X-rays are irradiated from the front of the lumbar vertebrae of the subject. In addition, in a DXA device, when the bone density of the proximal femur is measured, X-rays are irradiated from the front of the proximal femur of the subject. The ultrasound method is a method for measuring bone density by applying ultrasound to bones such as the heel and shin. Here, "the front of the lumbar vertebrae" and "the front of the proximal femur" are intended to be in the direction of correctly facing the imaging site such as the lumbar vertebrae and the proximal femur, and may be on the ventral side of the subject's body or on the back side of the subject. In the MD (micro densitometry) method, X-rays are irradiated onto the hand. In the ultrasound method, ultrasound is applied to bones such as the lumbar vertebrae, femur, heel, or shin to measure bone density. The image data does not have to be X-ray images, but may be any image that contains bone information. For example, it can be estimated from MRI (magnetic resonance imaging) images, CT (computed tomography) images, PET images, ultrasound images, etc.

[0035] In this embodiment, when a user accesses the analysis device 1A via the Internet, the communication control unit 11 may display a browser image 200 on a web page and transmit an encrypted application via the communication unit 15. When a user inputs image data 301, patient identification information 320, and attribute information 303 to be analyzed and clicks the "Send" button 204, the patient identification information 320 is encrypted, and the encrypted patient information 302 is transmitted to the analysis device 1A together with the image data 301 and attribute information 303. Therefore, the user does not need to perform encryption processing of the patient identification information 320. The encryption processing may be performed after the user clicks the "Send" button 204, rather than when the user clicks the button.

[0036] The encryption method may be a known method, and is not limited thereto. For example, an encryption method combining a public key and a private key may be used. The encryption may be a method that cannot be decrypted even by the operator of the image remote analysis system 70A. This reduces the risk that the combination of the image data 301 and the patient identification information 320 will be leaked to the operator, even if the user requests image analysis from the operator of the image remote analysis system 70A.

[0037] The acquisition unit 12 acquires the X-ray image data 300 , the encrypted patient information 302 , and the attribute information 303 transmitted from the user terminal 20 , and transmits them to the analysis unit 13 .

[0038] The analysis unit 13 inputs the X-ray image data 300 transmitted from the acquisition unit 12 to the machine model 131, and processes the output data from the machine model 131 as necessary to generate analysis result data 173. The generated analysis result data 173 is linked to at least the encrypted patient information 302 and the attribute information 303, and stored in the storage unit 17.

[0039] The communication control unit 11 acquires the analysis result data 173 linked to the encrypted patient information 302 and the attribute information 303 from the storage unit 17 , and transmits it to the user terminal 20 via the communication unit 15 .

[0040] When the user terminal 20 receives the analysis result data 173, the encrypted patient information 302 is automatically decrypted to generate patient identification information 320. The encrypted patient information 302 may be decrypted when the analysis result data 173 is transmitted, or the user may decrypt it by themselves.

[0041] This allows the user terminal 20 to display the analysis result data 173 on the screen together with the patient identification information 320 including the patient's name, identification number, or the like.

[0042] FIG. 7 is a schematic diagram showing an example of a browser image 400 displayed on the screen of the user terminal 20 that has received the analysis result data 173. The browser image 400 is an example of display image data generated by the generation unit 18. The words <Bone Density Analysis>, which indicate the content of the image analysis, are displayed at the top of the browser image 400. The words <Analysis Results>, which indicate that this is a screen for displaying the analysis results, are displayed in the upper left corner of the browser image 400. Below that, patient information may be displayed. Below that, the text "Your bone density is □ / cm 2 The text "Your femur fracture probability is □%" may be displayed below the box 401. The estimated bone density value is displayed in box 401. The text "Your femur fracture probability is □%" may be displayed below the box 402. The ratio to the Young Adult Mean (YAM) is displayed in box 402. The text "Your femur fracture probability is □%" may be displayed below the box 402. The text "Your femur fracture probability is □%" may be displayed below the box 402. The text "Judgment" may be displayed below the box 402. The text "Bone mass reduction" may be displayed in box 403. For example, if the ratio to the YAM is below 80%, it is judged as "bone mass reduction", and if it is below 70%, it is judged as "osteoporosis". The browser image 400 may have an "Exit" button 404 and a "Back" button 405. The "Back" button may be, for example, a button to return to the previous screen, a button to return to the home screen, or a button to return to a specific screen.

[0043] (Generation part 18) Next, a description will be given of the details of the generating unit 18. In this embodiment, the generating unit 18 generates derivation basis data 311 of the analysis result analyzed by the analyzing unit 13. The communication control unit 11 may transmit the derivation basis data 311 generated by the generating unit 18 to the user terminal 20.

[0044] Although the analysis result data 173 is based on the output from the machine model 131, the output from the machine model 131 does not include the analysis process. Therefore, it is generally not possible to judge the reliability of the output from the machine model 131 by looking at only the output. Therefore, transmitting the analysis result data 173 including the derivation basis data 311 to the user terminal 20 can help improve the user's sense of reliability in the analysis result.

[0045] For example, the derivation basis data 311 may be processed image data 310 (basis image data) in which new information is added to the image data 301 input to the machine model 131. For example, the new information may be information indicating an area that is the main basis for deriving the analysis result in the image data 301. The processed image may be an image in which information indicating the area that is the main basis is added to the input image data. The information indicating the area is, for example, information that indicates the range of the area, such as coloring or framing. In image analysis, since a certain area of ​​the image is often the main basis for estimation, the user can confirm the area that is the basis for estimation by using information indicating such an area. In this case, the display image displayed on the user terminal 20 includes a processed image in which the attention area information is added to the medical image input to the machine model 131. The generating unit 18 may generate a display image in which the medical image (image data 301) and the processed image are arranged side by side. Such a display image makes it easy for the user to compare the original image that has been analyzed with the processed image. Alternatively, the control unit 16 may switch between the medical image and the processed image by a user operation and display them. For example, the control unit 16 may alternately display the medical image and the processed image when the user clicks a switching button.

[0046] It should be noted that the processed image data 310 does not need to include all of the information of the image data 301 input by the user. For example, the processed image data 310 may be image data obtained by trimming the input image data 301, or may be an image with a lower resolution than the input image data 301. In this way, the amount of data to be transmitted and received can be reduced.

[0047] 8 is a schematic diagram showing an example of a browser image 500 that transmits derived basis data 311 to the user terminal 20 in addition to the analysis result data 173. The browser image 500 is an example of display image data generated by the generation unit 18. In the browser image 500, the derived basis data 311 (processed image data 502 including an area 503) is added to the analysis result data 173 shown in FIG.

[0048] Specifically, the browser image 500 displays processed image data 502 obtained by additionally processing the input image data (X-ray image data 300 in FIG. 6) with a rectangular area 503, together with the analysis result 501. As shown in FIG. 8, in this embodiment, the analysis result 501 derived by the machine model 131 that analyzes the image data, and the processed image data 502 including the derivation basis data 311 are displayed on one screen. The browser image 500 may display a "Back" button 506, an "Exit" button 507, and a "Future Prediction" button 505. The role of the "Future Prediction" button 505 will be described later.

[0049] FIG. 9 is an enlarged schematic diagram of the image of region 503 in FIG. 8. Region 503 includes four lumbar vertebrae, L1 to L4. This indicates that the lumbar vertebrae L1 to L4 are the region on which the analysis results are based. In fact, it is known that the bone density of the lumbar vertebrae L1 to L4 is related to the average bone density of the whole body. In other words, the enclosed region 503 indicates that the analysis results of the mechanical model 131 are derived based on the bone density of these lumbar vertebrae L1 to L4.

[0050] The region of interest on which the analysis result is based may include a region in which the medical image is segmented. Segmentation means dividing an image into several regions. Segmentation is performed to reduce the amount of analysis processing of the mechanical model 131. That is, the mechanical model 131 may analyze only the segmented region. The segmented region may be set to any range. The segmented region may be, for example, rectangular, square, or circular. When the segmented region is a square, the amount of analysis processing of the mechanical model 131 can be reduced. When analyzing an X-ray image of the lumbar vertebrae, for example, a range including the lumbar vertebrae L1 to L4 is segmented. Therefore, the size of the segmentation region may change depending on the size of the lumbar vertebrae L1 to L4 in the image. The segmentation region may be set, for example, slightly larger than the lumbar vertebrae L1 to L4 in the arrangement direction of the lumbar vertebrae L1 to L4, or may be set so as to overlap with the sides of the lumbar vertebrae L1 to L4. The segmentation region may always have a predetermined size, or may have a size set according to the medical image. For example, the segmentation region may specify the positions of the lumbar vertebrae L1-L4, set the lengths of the lumbar vertebrae L1-L4 in the arrangement direction, and then set the lengths of the lumbar vertebrae L1-L4 in the vertical direction. The segmentation may be performed by the machine model 131. The machine model 131 may learn images with annotations of the analysis region in order to perform the segmentation.

[0051] The generating unit 18 may also generate a heat map of the region on which the analysis result is based. In this case, for example, the outer edge of the heat map indicates the segmentation region. The heat map is a method of expressing the magnitude of bone density with the density of any color. For example, the generating unit 18 may generate a heat map indicating the degree of attention. Also, the generating unit 18 may generate a heat map indicating the numerical value of bone density. Also, the generating unit 18 may generate a heat map indicating the possibility (probability) of fracture. The processed image may be an image in which a heat map indicating the information on the region of interest is superimposed on a medical image. The image used for the heat map may be a still image or a video. By showing it as a video, for example, by fading various heat maps in order, it becomes easy to visually recognize the relationship between each heat map. Also, when the analysis result is a heat map of bone density including regions other than the segmentation region, a part of the segmentation region may be surrounded by a frame.

[0052] The generating unit 18 may obtain information on the region on which the analysis result is based from the analyzing unit 13. Specifically, the generating unit 18 may obtain the region on which the analysis result is based from the analyzing unit 13, and generate information indicating the region (such as a frame surrounding the region 503). The generating unit 18 may obtain information indicating the degree of attention data, bone density data, or the possibility of fracture within the region from the analyzing unit 13, and generate a heat map. The generating unit 18 may generate any one or more of a heat map indicating the degree of attention data, a heat map indicating the bone density data, and a heat map indicating the possibility of fracture. When the heat map indicating the degree of attention data, the heat map indicating the bone density data, and the heat map indicating the possibility of fracture are displayed in an overlapping manner, the generating unit 18 may cause the color of the heat map indicating the degree of attention data to be different from the color of the heat map indicating the bone density data, and the color of the heat map indicating the possibility of fracture.

[0053] The machine model 131 analyzes, for example, the X-ray image data 300 using a neural network model (NNM). In the NNM, the image is divided into small regions, each of which is quantified, and the multiple regions are pooled to be integrated into a larger region, which is then quantified again. Therefore, the machine model 131 may extract regions having values ​​(e.g., relatively large values) that affect the processing results as regions that serve as the basis.

[0054] 8, processed image data 502 is described in which region 503 that is the basis of the analysis result is superimposed on X-ray image data 300 to be analyzed. However, the processed image data is not limited to this. For example, analysis device 1 may transmit position information (such as coordinates) of the region that is the basis of the analysis result in the image to be analyzed to a user terminal, and cause the user terminal to display the processed image data with the region displayed on the image to be analyzed.

[0055] A screen like the one shown in Fig. 8 can be used when a doctor explains the analysis results to a patient. In other words, it is possible to not only simply explain the analysis results to the patient, but also to explain which area of ​​the image data was the basis for the analysis results. This not only improves the doctor's confidence in the analysis results, but also makes it easier for the patient to accept the analysis results.

[0056] In the example shown in FIG. 8, the analysis result 501 and the processed image data 502 including the derivation basis data 311 are displayed on one screen. However, "one screen" does not have to be displayed on the screen at the same time. For example, the screen may be scrolled up and down or left and right to display. In other words, the range displayed by scrolling the screen up and down or left and right is referred to as "one screen."

[0057] FIG. 10 is a schematic diagram showing an example of a browser image 700 that displays the part of a bone that is predicted to fracture at the present time of the patient, estimated from the image data analyzed by the analysis unit 13, the possibility of the part being fractured, and the possibility of the part being fractured in three years' time of the patient. Note that the future time is not limited to three years' time, but may be any time (e.g., X years' time). The browser image 700 is an example of display image data generated by the generation unit 18. This is displayed by clicking the "future prediction" button 505 at the bottom right of the browser image 500 shown in FIG. 8.

[0058] Browser image 700 displays the words <Bone Density Analysis> at the top, indicating the subject of image analysis. In the upper left corner of browser image 700, the words <Future Prediction> are displayed, indicating that this is a screen for displaying future predictions. Below that, patient information may be displayed. Below that, the words "Your femur fracture probability is □%" are displayed. Box 701 displays the currently estimated femur fracture probability. Further below that, the words "Your femur fracture probability in 3 years is □%" may be displayed. Box 702 displays the predicted fracture probability in 3 years. This image may also display information regarding the basis for the estimation or prediction.

[0059] Fig. 11 is a diagram showing an example of a browser image 800 in which derived basis data has been added to the display image shown in Fig. 10. The browser image 800 is an example of display image data generated by the generation unit 18. As shown in Fig. 11, the browser image 800 displays an area 801 displaying the femur fracture possibility at the present time and three years from now, which is derived by the machine model 131, as well as an image 802 showing the derived basis data.

[0060] Such a browser image 800 can be used when a doctor explains the current and future possibility of fracture to a patient. When information on the basis of estimation or prediction is also displayed, the effect of increasing persuasiveness to the patient can be obtained.

[0061] (Machine Model 131) Next, the machine model 131 will be described. The machine model 131 is a model that estimates the bone condition of a subject, and input image data is an image including bones, and outputs an estimation result regarding the bone condition as an analysis result. For example, the machine model 131 is a trained model that has been trained to output an estimation result or a calculation result regarding the bone condition, such as bone density, a relative comparison of bone density, and a possibility of fracture, from an X-ray image of a bone. The bone density may be a calculated bone density of a bone part included in the image data, or may be an average bone density of the whole body estimated from the image data. A known method can be used to calculate the bone density from an image. The relative comparison of bone density is the ratio of the estimated bone density to the YAM. The possibility of fracture is the possibility of a bone at a specific part (for example, the femoral neck, etc.) being fractured. The estimation result regarding the bone condition may be an estimation result at the time when the image is taken, or may be a prediction at a time when a predetermined period has passed since that time.

[0062] The mechanical model 131 may output as an estimation result at least one of the following: bone density estimated at the time when the image data is captured, bone density predicted when a specified period of time has passed since the image data is captured, fracture site and possibility estimated at the time when the image data is captured, and fracture site and possibility predicted when a specified period of time has passed since the image data is captured.

[0063] (Mechanical Model 131 Learning Methods) Next, a learning method of the machine model 131 will be described. The learning of the machine model 131, for example, learning to estimate bone density, may be performed using an X-ray image of a bone with a specified bone density as training data. Learning to estimate the possibility of a fracture may be performed using an X-ray image of a bone and data on whether the patient subsequently had a fracture within a specified period as training data. Relative comparison of bone density does not need to be learned, and is obtained by dividing the estimated bone density by YAM. Furthermore, learning of future prediction may be performed using an X-ray image of a bone with a specified bone density and data on how much bone density the patient had after a specified period or whether the patient subsequently had a fracture as training data. By including lifestyle habits such as the amount of exercise, diet, smoking, and drinking of the patient as training data in the training data, the machine model 131 capable of more accurate estimation or prediction can be constructed.

[0064] In the second embodiment, the mechanical model 131 has been described by taking as an example a mechanical model that has been trained to estimate bone conditions such as bone density. In this case, the input image data is an X-ray image of a bone, and the output is an estimation result regarding the bone condition. However, the mechanical model 131 is not limited to such a model. For example, the mechanical model 131 may be a cytopathological analysis model. In this case, the input medical image may be a microscopic image of a subject's cells, and the analysis result may be information regarding pathological mutations in the cells.

[0065] FIG. 12 is a schematic diagram showing an example of a browser image 900, which is a display image to which a cytopathological analysis result and its derivation basis data are added when the machine model 131 is a cytopathological analysis model. The browser image 900 is an example of display image data generated by the generation unit 18. In the browser image 900 shown in FIG. 12, the characters <cytopathological analysis> indicating the contents of the image analysis are displayed at the top. In the upper left corner of the browser image 900, the characters <judgment result> indicating that the screen displays the result of judging whether the cell is benign or malignant are displayed. Below that, patient information may be displayed. In the area 901 below that, the characters "The possibility of malignancy is □%" are displayed. In the box, a numerical value of the possibility that the cell is malignant is displayed. Furthermore, to the right of the area 901, an image 902 is displayed in which an area 903, which is the main basis for deriving the judgment result, is added to the analyzed image as the derivation basis data. The area 903 is an area including cells judged by the machine model 131 to be malignant. Furthermore, the browser image 900 may display an “Exit” button 904 and a “Back” button 905 .

[0066] Moreover, one screen may be a screen displayed by screen scrolling. FIG. 13 is a schematic diagram showing an example of displaying a display image on one screen by scrolling. An area 1301 shown by a solid line in FIG. 13 is a display area of ​​the user terminal 20 (display device). An area 1302 shown by a dotted line is one screen. In the upper part of the area 1302, data of the analysis result 501 of bone density described in FIG. 8 is displayed. In the lower part of the area 1302, processed image data 502, which is the basis derivation data described in FIG. 8, is displayed. When the area 1302 is displayed for the first time, the data of the analysis result 501 of bone density is displayed in the area 1301 as shown in the upper part of FIG. 13. However, when the user scrolls the area 1302 up, the processed image data 502 appears as shown in the upper part of FIG. 13. Even with this configuration, the analysis result and its derived basis data can be displayed by the user simply scrolling the screen.

[0067] According to the configuration of the image remote analysis system 70A according to the second embodiment, the derivation basis data 311 can be provided to the user together with the analysis result data 173. This has the effect of improving the user's sense of reliability in the analysis result. In addition, it has the effect of making the patient more likely to understand the analysis result when explaining it to the patient.

[0068] Next, a description will be given of the flow of an image remote analysis method S2 executed by the control unit 16 according to the embodiment 2. The image remote analysis method S2 includes steps S21 to S24 (not shown).

[0069] When a user accesses the analysis device 1A, the communication control unit 11 displays a browser image 200, which is an input screen for inputting acquired information 172 including image data 301 to be analyzed, patient identification information 320, and attribute information 303, on a web page via the communication unit 15 (S21).

[0070] The acquisition unit 12 acquires encrypted patient information 302 in which image data 301, attribute information 303, and patient identification information 320 included in the acquired information 172 input to the browser image 200 and transmitted from the user terminal 20 are encrypted (S22).

[0071] The analysis unit 13 analyzes the image data 301 sent from the acquisition unit 12 using the machine model 131 (S23).

[0072] This is a step in which the communication control unit 11 transmits the analysis result data 173 analyzed by the analysis unit 13, the derivation basis data 311, and the encrypted patient information 302 to the user terminal 20 (S24).

[0073] According to the above remote image analysis method S2, the derivation basis data 311 can be provided to the user together with the analysis result data 173. This has the effect of improving the user's sense of trust in the analysis result. In addition, it has the effect of making the patient more likely to understand the analysis result when explaining it to the patient.

[0074] [Software implementation example] The functions of image remote analysis system 70A (hereinafter referred to as the "system") are programs for causing a computer to function as the system, and can be realized by programs for causing a computer to function as each part of the system.

[0075] In this case, the system includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0076] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0077] In addition, some or all of the functions of each of the above units can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the above units is formed is also included in the scope of the present disclosure. In addition, the functions of each of the above units can be realized by, for example, a quantum computer.

[0078] The invention according to the present disclosure has been described above based on the drawings and examples. However, the invention according to the present disclosure is not limited to the above-mentioned embodiments. In other words, the invention according to the present disclosure can be modified in various ways within the scope of the present disclosure, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments are also included in the technical scope of the invention according to the present disclosure. In other words, it should be noted that a person skilled in the art can easily make various modifications or corrections based on the present disclosure. It should also be noted that these modifications or corrections are included in the scope of the present disclosure.

[0079] In the present disclosure, a medical image is analyzed and the medical image is changed based on the region of interest information related to the region of interest that is focused on during the process of outputting the analysis result, but the present disclosure is not limited thereto. For example, analysis may be performed using a trained machine model with only numerical values ​​and the following information data as input information without including a medical image. In this case, the element of interest information that is focused on during the process of outputting the analysis result may be displayed on the display image together with the analysis result. The element of interest information may also be information that highlights a part of the input information. For example, when age and gender information are the elements of interest, the items or numerical values ​​of age and gender may be highlighted in a conspicuous color.

[0080] Input information may include age, sex, weight, height, whether or not a fracture has occurred, the location of the fracture, fracture history, family members' (e.g., parents') history of fractures, underlying diseases (e.g., food and / or drug allergies, diseases related to the development of osteoporosis and diseases unrelated to the development of osteoporosis, etc.), smoking history, drinking habits (e.g., drinking frequency and amount, etc.), occupational history, exercise history, medical history (e.g., history of bone disease), menstruation (e.g., cycle and presence / absence, etc.), menopause (e.g., possibility and presence / absence, etc.), artificial joints (e.g., type, presence / absence of spinal implant or knee joint, and timing of replacement surgery), blood test results, urine test results, medications being taken, gene sequences, etc.

[0081] (summary) (Aspect 1) The image generating device according to aspect 1 of the present disclosure includes an acquisition unit that acquires analysis results output from a machine model that analyzes a medical image of a subject and area of ​​interest information relating to an area of ​​interest within the medical image that is focused on during the process of outputting the analysis results, and an image generating unit that generates a display image by modifying the medical image based on the area of ​​interest information.

[0082] (Aspect 2) An image generating device according to a second aspect of the present disclosure is in accordance with the first aspect above, wherein the display image includes a processed image in which the area of ​​interest information is added to the medical image input to the machine model.

[0083] (Aspect 3) An image generating device according to aspect 3 of the present disclosure is the above aspect 1 or 2, wherein the region of interest includes a region into which the medical image is segmented.

[0084] (Aspect 4) The image generating device according to a fourth aspect of the present disclosure is any one of the first to third aspects, wherein the region of interest information is displayed as a heat map.

[0085] (Aspect 5) An image generating device according to a fifth aspect of the present disclosure is the image generating device of the second aspect, wherein the processed image is an image onto which the region-of-interest information is superimposed.

[0086] (Aspect 6) The image generating device according to aspect 6 of the present disclosure is in aspect 2 or 5 above, wherein the image generating unit generates the display image in which the medical image and the processed image are arranged side by side.

[0087] (Aspect 7) The image generating device according to a seventh aspect of the present disclosure is any one of the second, fifth, and sixth aspects, in which the display image and the processed image are interchangeable by an operation.

[0088] (Aspect 8) An image generating device according to aspect 8 of the present disclosure is any one of aspects 1 to 7 above, wherein the mechanical model is a model that makes an estimation regarding the condition of the subject's bones, the medical image is an image showing the subject's bones, and the analysis result includes an estimation result regarding the condition of the subject's bones.

[0089] (Aspect 9) An image generating device according to a ninth aspect of the present disclosure is any one of the first to eighth aspects, in which the display image is displayed as a heat map in which the analysis result and the area of ​​interest information are different aspects.

[0090] (Aspect 10) In an image generating device according to aspect 10 of the present disclosure, in the above aspect 8, the estimation result is at least one of an estimation of the fracture site of the subject and the possibility that the subject will suffer a fracture.

[0091] (Aspect 11) An image generating device according to aspect 11 of the present disclosure is any one of aspects 1 to 7 above, wherein the mechanical model is a cell pathology analysis model, the medical image is a microscopic image of the subject's cells, and the analysis results include information regarding pathological mutations in the subject's cells.

[0092] (Aspect 12) The image generating device according to a twelfth aspect of the present disclosure is any one of the first to eleventh aspects, wherein the display image is displayed on one screen.

[0093] (Aspect 13) The image generation method according to aspect 13 of the present disclosure includes an acquisition step of acquiring analysis results output from a machine model that analyzes a medical image of a subject and area of ​​interest information relating to an area of ​​interest within the medical image that is focused on during the process of outputting the analysis results, and an image generation step of generating a display image in which the medical image is modified based on the area of ​​interest information.

[0094] (Aspect 14) A program according to aspect 14 of the present disclosure is an image generation program for causing a computer to function as the image generation device described in claim 1, and is an image generation program for causing a computer to function as the acquisition unit and the image generation unit.

[0095] (Aspect 15) A recording medium according to a fifteenth aspect of the present disclosure is a computer-readable non-transitory recording medium having the image generating program according to claim 14 recorded thereon. [Explanation of symbols]

[0096] 70, 70A···Image remote analysis system 1A...Analysis device 3. Image generation device 11,33 Communication control section 12...Acquisition part 13...Analysis department 131... Machine Model 14 Analysis result transmission section 15,50···Communications Department 16,30...Control section 17,40...Storage section 18...Generation part 19... Judgment section 20 User terminal 31...Image generation unit 32 Data acquisition unit (acquisition unit) 41. Analysis Data 42. Derivation basis data 43 Display image data 60...Analysis equipment 601... Machine Model 70...Display device

Claims

1. an acquisition unit that acquires an analysis result on a bone condition output based on a mechanical model from input information including an X-ray image showing the bones of a subject, and information on a region of interest that includes at least a segmented region that is at least a part of the X-ray image; an image generating unit that generates a display image by modifying the X-ray image based on the region of interest information; the display image is an image including the analysis result and the region of interest information, the display image includes a processed image in which the region of interest information is added to the X-ray image input to the machine model, The X-ray image and the processed image can be interchanged by operation. Image generation system.

2. An acquisition unit that acquires analysis results regarding the state of bones output based on a mechanical model from input information including an X-ray image showing the bones of a subject, and area of ​​interest information regarding an area of ​​interest that includes at least a segmented region that is at least a part of the X-ray image; an image generating unit that generates a display image by modifying the X-ray image based on the region of interest information; the display image is an image including the analysis result and the region of interest information, The display image shows the analysis result and the attention area information in a heat map having different aspects. Image generation system.

3. The image generating system of claim 1 or 2, wherein the region of interest information includes at least a portion of a bone of the subject in the segmented region.

4. The image generating system of claim 1 or 2, wherein the region of interest information includes at least a portion of a lumbar vertebra of the subject in the segmented region.

5. The image generating system of claim 1 or 2, wherein the segmented regions are rectangular.

6. The image generating system according to claim 2 , wherein the display image includes a processed image in which the region of interest information is added to the X-ray image input to the machine model.

7. The image generating system according to claim 1 or 2, wherein the region of interest includes all of the regions into which the X-ray image is segmented.

8. The image generating system according to claim 1 or 2, wherein the region of interest information is displayed as a heat map.

9. The image generating system according to claim 1 , wherein the processed image is an image on which the region-of-interest information is superimposed.

10. The image generating system according to claim 1 , wherein the image generating unit generates the display image in which the X-ray image and the processed image are arranged side by side.

11. The image generating system according to claim 6 , wherein the X-ray image and the processed image are interchangeable by operation.

12. The image generation system of claim 1 or 2, wherein the machine model performs an analysis of segmented regions of the X-ray image.

13. The image generating system according to claim 1 , wherein the region of interest is a region that is focused on in a process of outputting the analysis result.

14. The image generating system according to claim 1 , wherein the display image shows the analysis result and the region of interest information in a heat map in a different manner.

15. The analysis results include an estimation result regarding a bone condition of the subject, The image generating system according to claim 1 or 2, wherein the estimation result is at least one of an estimation result of a fracture site of the subject and a possibility that the subject will have a fracture.

16. The analysis results include an estimation result regarding a bone condition of the subject, The image generating system according to claim 1 or 2, wherein the estimation result includes an analysis result regarding a bone density of the subject.

17. The image generation system of claim 2, wherein the displayed image includes a processed image in which the area of ​​interest information has been added to the X-ray image input to the mechanical model, and the processed image is an image on which the area of ​​interest information is superimposed.

18. The image generation system of claim 2, wherein the display image includes a processed image in which the area of ​​interest information has been added to the X-ray image input to the mechanical model, and the image generation unit generates the display image in which the X-ray image and the processed image are juxtaposed.

19. an acquisition step of acquiring an analysis result on a bone condition output based on a machine model from input information including an X-ray image showing the bones of a subject, and region of interest information on a region of interest that includes at least a segmented region that is at least a part of the X-ray image; and generating a display image by modifying the X-ray image based on the region of interest information. the display image is an image including the analysis result and the region of interest information, the display image includes a processed image in which the region of interest information is added to the X-ray image input to the machine model, The X-ray image and the processed image can be interchanged by operation. Image generation method.

20. An acquisition step for acquiring analysis results regarding the state of bones output based on a mechanical model from input information including an X-ray image showing the bones of a subject, and area of ​​interest information regarding an area of ​​interest that includes at least a portion of a segmented region that is at least a part of the X-ray image; and generating a display image by modifying the X-ray image based on the region of interest information. the display image is an image including the analysis result and the region of interest information, The display image shows the analysis result and the attention area information in a heat map having different aspects. Image generation method.

21. 3. An image generating program for causing a computer to function as the image generating system according to claim 1, the image generating program causing a computer to function as said acquisition unit and said image generating unit.

22. A non-transitory computer-readable recording medium having the image generating program according to claim 21 recorded thereon.

Images