Ultrasound treatment and diagnosis system, program, and affected area tracking evaluation method

Abstract

To achieve possibilities of improvement of tracking of an organ under the existence of the acoustic shadow and continuous diagnosis.SOLUTION: An ultrasonic treatment diagnostic system used in the treatment and / or diagnosis using an ultrasonic image comprises: an image acquisition unit; an image group storage unit which stores a plurality of ultrasonic images; an organ mask generation unit which generates an organ mask corresponding to an organ included in the ultrasonic image; an acoustic shadow mask generation unit which generates an acoustic shadow mask corresponding to the acoustic shadow included in the ultrasonic image; a complementation object image setting unit which sets a complementation object image being the ultrasonic image including the acoustic shadow and requiring complementation; a reference image search unit which searches for a reference image being the ultrasonic image suitable to complementation of the acoustic shadow included in the complementation object image from the plurality of ultrasonic images that are stored; a complementation region extraction unit which extracts a complementation region being a region used for complementation of the acoustic shadow included in the complementation object image from the reference image; and a composite complementation image generation unit which performs composite complementation on the acoustic shadow included in the complementation object image in the complementation region.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an ultrasonic diagnostic and treatment system. , P The present invention relates to a program and a method for evaluating the tracking of affected areas. [Background technology]

[0002] High Intensity Focused Ultrasound (HIFU) is a treatment method that focuses ultrasound at one point to cauterize the affected area. This treatment method is used to treat stones and tumors in organs. The advantage of this treatment is that it can be performed without opening the abdomen with a scalpel, which reduces the burden on the patient's body. In addition, this treatment is an ultrasound version of radiation therapy, so there is no risk of exposure like with radiation therapy, and it can be said to be non-invasive. However, a problem with this treatment is that organs move due to the patient's breathing, heartbeat, etc., making it difficult to focus precisely on the affected area. If the focus cannot be precisely focused on the affected area, there is a risk of damaging normal tissue. Therefore, a system is needed that can treat the affected area while tracking the respiratory movement.

[0003] The Non-Invasive Ultrasound Theragnostic System (NIUTS) that the inventors have been researching is a system that extracts, tracks, and monitors affected areas that actively move due to breathing, pulsation, etc., while focusing ultrasound waves and irradiating the affected areas with pinpoint accuracy, thereby cauterizing cancer tissue and breaking up stones non-invasively and with little burden on the patient, without making incisions on the surface of the skin.

[0004] Problems with the system that the present inventors have been researching include strong reflection of sound waves from hard tissues such as bones, etc. In particular, the kidney, which is the target of HIFU treatment, is covered by the ribs and sternum, so that acoustic shadows are generated, and there is a risk of missing information about the deeper part. The effects of acoustic shadowing include the difficulty of diagnosis and monitoring because the affected area of ​​the HIFU treatment is hidden, and the decrease in accuracy of tracking of organs.

[0005] Also, there is known an in vivo motion tracking device that aims to improve the accuracy and robustness of tracking a part of interest moving in a living body (see, for example, Patent Document 1). In the technology described in Patent Document 1, an organ position estimation unit obtains an estimated position of an organ in a living body image based on the past movement of the organ. Also, in the technology described in Patent Document 1, when an acoustic shadow overlaps a part of an organ, the position and rotation angle of the organ are estimated based on a non-missing part of the organ's contour. Incidentally, in the technique described in Patent Document 1, in order to complement an ultrasound image of an organ that includes an acoustic shadow, the contour of the organ must be estimated. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2018-143416 A Summary of the Invention [Problem to be solved by the invention]

[0007] In view of the above, the present invention provides an ultrasonic diagnostic and treatment system that can improve the tracking of organs and enable continuous diagnosis even in the presence of acoustic shadows. , P The object of the present invention is to provide a program and a method for evaluating tracking of the affected area. In particular, the present invention provides an ultrasonic diagnostic and treatment system capable of complementing an acoustic shadow contained in an ultrasonic image of an organ without the need to estimate the contour of the organ. , P The object of the present invention is to provide a program and a method for evaluating tracking of the affected area. [Means for solving the problem]

[0008] Through intensive research, the present inventors have discovered an image completion algorithm for acoustic shadow regions that utilizes the fact that organs move due to breathing and the like. In detail, the inventors have found that even if a particular part of an organ is affected by acoustic shadowing in an ultrasound image of the organ taken at a certain point in time because the organ moves due to breathing, etc., that part of the organ is not affected by acoustic shadowing in an ultrasound image of the organ taken at another point in time, and therefore, by using an ultrasound image of an organ in which that part is not affected by acoustic shadowing, it is possible to complement an ultrasound image of an organ in which that part is affected by acoustic shadowing.

[0009] One aspect of the present invention is an ultrasound treatment and diagnosis system used for treatment and / or diagnosis using ultrasound images, comprising an image acquisition unit that acquires ultrasound images, an image group storage unit that stores a plurality of ultrasound images acquired by the image acquisition unit, an organ mask generation unit that generates an organ mask corresponding to an organ included in the ultrasound images acquired by the image acquisition unit, an acoustic shadow mask generation unit that generates an acoustic shadow mask corresponding to an acoustic shadow included in the ultrasound images acquired by the image acquisition unit, a complementation target image setting unit that sets a complementation target image, which is an ultrasound image acquired by the image acquisition unit, which includes an acoustic shadow and requires complementation, and a selection unit that selects an ultrasound image suitable for complementing the acoustic shadow included in the complementation target image from the plurality of ultrasound images stored by the image group storage unit. a complementation area extraction unit that extracts from the reference image a complementation area that is an area used to complement an acoustic shadow included in the image to be complemented; and a synthetic complemented image generation unit that synthesizes and complements the acoustic shadow included in the image to be complemented with the complementation area extracted by the complementation area extraction unit, wherein the reference image search unit searches for the reference image based on an organ mask corresponding to an organ included in the image to be complemented, an acoustic shadow mask corresponding to the acoustic shadow included in the image to be complemented, an organ mask corresponding to an organ included in reference candidate images, which are each a plurality of ultrasound images stored by the image group storage unit, and an acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image.

[0010] In the ultrasound diagnostic and treatment system according to one aspect of the present invention, the organ mask generating unit may generate the organ mask by performing organ segmentation using a segmentation method for medical images.

[0011] In the ultrasound diagnostic and treatment system according to one aspect of the present invention, the acoustic shadow mask generating unit may generate the acoustic shadow mask by performing acoustic shadow segmentation using a segmentation technique for medical images.

[0012] In an ultrasound therapeutic diagnosis system of one embodiment of the present invention, the reference image search unit may search for the reference image based on the similarity between an organ mask corresponding to an organ included in the image to be complemented and an organ mask corresponding to an organ included in the reference candidate image.

[0013] In an ultrasound treatment diagnosis system according to one embodiment of the present invention, the reference image search unit may calculate the similarity between an organ mask corresponding to an organ included in the image to be complemented and an organ mask corresponding to an organ included in the reference candidate image by using a Dice coefficient.

[0014] In an ultrasound treatment diagnosis system according to one embodiment of the present invention, the reference image search unit may search for the reference image based on the similarity between an acoustic shadow mask corresponding to the acoustic shadow included in the image to be complemented and an acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image.

[0015] In an ultrasound treatment and diagnosis system according to one embodiment of the present invention, the reference image search unit may calculate a similarity between an acoustic shadow mask corresponding to an acoustic shadow included in the image to be complemented and an acoustic shadow mask corresponding to an acoustic shadow included in the reference candidate image by using a Dice coefficient.

[0016] In one embodiment of the ultrasound therapeutic diagnosis system of the present invention, the reference image search unit aligns the organ mask corresponding to the organ included in the reference candidate image and the acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image with the organ mask corresponding to the organ included in the image to be complemented and the acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image, so that the organ contours of the organ mask corresponding to the organ included in the image to be complemented and the organ mask corresponding to the organ included in the reference candidate image match, and when the organ contours of the organ mask corresponding to the organ included in the image to be complemented and the organ mask corresponding to the organ included in the reference candidate image match, and the reference candidate image is selected as the reference image when the acoustic shadow mask corresponding to the acoustic shadow included in the image to be complemented and the acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image do not overlap.

[0017] In one embodiment of the ultrasound therapeutic diagnosis system of the present invention, the reference image search unit may search for the reference image by using another ultrasound image among the multiple ultrasound images stored by the image group storage unit as the reference candidate image when the contours of the organs in the organ mask corresponding to the organ included in the image to be complemented and the organ mask corresponding to the organ included in the reference candidate image match, and the contours of the acoustic shadows in the acoustic shadow mask corresponding to the acoustic shadow included in the image to be complemented and the acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image at least partially overlap.

[0018] In one embodiment of the ultrasound therapeutic diagnosis system of the present invention, the reference image search unit may perform at least one of translation, rotation, enlargement, and reduction of the organ mask corresponding to the organ included in the reference candidate image and the acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image, in order to match the contours of the organs in the organ mask corresponding to the organ included in the image to be complemented and the organ mask corresponding to the organ included in the reference candidate image.

[0019] In an ultrasound treatment and diagnosis system according to one embodiment of the present invention, the reference image search unit may use a Dice coefficient to evaluate whether or not the contours of an organ match between an organ mask corresponding to an organ included in the image to be complemented and an organ mask corresponding to an organ included in the reference candidate image.

[0020] In the ultrasound diagnostic and treatment system according to one aspect of the present invention, the shape of the complement region extracted from the reference image by the complement region extraction unit may be identical to the shape of an acoustic shadow included in the image to be complemented.

[0021] In one embodiment of the ultrasound therapeutic diagnosis system of the present invention, the shape of the complementation region extracted from the reference image by the complementation region extraction unit may be different from the shape of the acoustic shadow contained in the image to be complemented, the complementation region may be smaller than the acoustic shadow contained in the image to be complemented, and the complementation region may include only an area corresponding to a part inside the contour of an organ.

[0022] An ultrasound therapeutic diagnosis system of one embodiment of the present invention includes a composite complement image display unit that displays a composite complement image generated by the composite complement image generation unit, and the composite complement image display unit may display the complement area contained in the composite complement image in a distinguishable manner from areas other than the complement area.

[0023] In the ultrasound diagnostic and treatment system according to one aspect of the present invention, the ultrasound images acquired by the image acquisition unit may include ultrasound images obtained using a three-dimensional probe.

[0025] One aspect of the present invention includes a computer that performs an image acquisition step of acquiring an ultrasound image, an image group storage step of storing a plurality of ultrasound images acquired in the image acquisition step, an organ mask generation step of generating an organ mask corresponding to an organ included in the ultrasound image acquired in the image acquisition step, an acoustic shadow mask generation step of generating an acoustic shadow mask corresponding to an acoustic shadow included in the ultrasound image acquired in the image acquisition step, a complement target image setting step of setting a complement target image which is an ultrasound image acquired in the image acquisition step, which includes an acoustic shadow and requires complementation, and a search for a reference image which is an ultrasound image suitable for complementing the acoustic shadow included in the complement target image from among the plurality of ultrasound images stored in the image group storage step. a reference image search step for extracting a complementation area from the reference image, which is an area used to complement the acoustic shadow contained in the image to be complemented, and a synthetic complement image generation step for synthetically complementing the acoustic shadow contained in the image to be complemented with the complementation area extracted in the complementation area extraction step, wherein in the reference image search step, the reference image is searched for based on an organ mask corresponding to the organ contained in the image to be complemented, an acoustic shadow mask corresponding to the acoustic shadow contained in the image to be complemented, an organ mask corresponding to the organ contained in reference candidate images, which are each a plurality of ultrasound images stored in the image group storage step, and an acoustic shadow mask corresponding to the acoustic shadow contained in the reference candidate image.

[0026] One aspect of the present invention is a method for evaluating tracking of an affected area, which uses template matching on a synthetic complement image generated by the synthetic complement image generation unit of the ultrasound treatment diagnosis system to evaluate the tracking of an affected area contained in the synthetic complement image. Effect of the Invention

[0027] According to the present invention, an ultrasonic diagnostic and treatment system capable of improving the tracking of organs and enabling continuous diagnosis in the presence of acoustic shadows is provided. , PA program and a method for evaluating tracking of the affected area can be provided. In particular, according to the present invention, there is provided an ultrasonic diagnostic and treatment system capable of complementing an acoustic shadow contained in an ultrasonic image of an organ without the need to estimate the contour of the organ. , P A program and a method for evaluating tracking of the affected area can be provided. [Brief description of the drawings]

[0028] [Figure 1] 1 is a diagram illustrating an example of an ultrasound diagnostic treatment system according to a first embodiment. [Diagram 2] 3A to 3C are diagrams showing examples of ultrasound images acquired by an image acquisition unit. [Diagram 3] 1 is a diagram showing an example of a process in which ultrasound images obtained using an ultrasound probe are acquired by an image acquisition unit and stored in an image group storage unit. [Figure 4] 13A and 13B are diagrams showing a comparison between an example in which an organ mask is generated by the organ mask generating unit using handwritten material and an example in which an organ mask is automatically generated by the organ mask generating unit. [Diagram 5] FIG. 13 is a diagram for explaining an organ mask generation experiment conducted by the present inventors. [Figure 6] FIG. 13 is a diagram for explaining the experimental conditions in an organ mask generation experiment conducted by the present inventors. [Figure 7] FIG. 13 is a diagram for explaining the results of an organ mask generation experiment conducted by the present inventors. [Figure 8] 11A and 11B are diagrams for explaining an example of alignment performed by a reference image search unit; [Figure 9] 11 is a diagram for explaining an example of extraction of a complemented region from a reference image by a complemented region extraction section and synthesis and complementation of a complement target image by a composite complemented image generation section (generation of a composite complemented image). FIG. [Figure 10] FIG. 1 is a diagram for explaining a synthesis and complementation experiment conducted by the present inventors. [Figure 11] FIG. 1 is a diagram for explaining an evaluation method of a synthesis complementation experiment conducted by the present inventors. [Figure 12]FIG. 11 is a diagram showing the results of a synthesis and complementation experiment (image synthesis experiment) conducted by the present inventors on the kidney cross-sectional image A shown in FIG. [Figure 13] 4 is a flowchart showing an example of processing executed in the ultrasound diagnostic treatment system of the first embodiment. [Figure 14] 11 is a diagram for explaining a method for evaluating the tracking of an affected area included in a synthetic complement image in a synthetic complement experiment conducted by the present inventors. FIG. [Figure 15] FIG. 13 is a diagram showing the results of a synthesis and complementation experiment (image synthesis experiment) conducted by the present inventors on a kidney cross-sectional image A (moving image A) using the ultrasound diagnostic and treatment system of the second embodiment. [Figure 16] FIG. 13 is a diagram for explaining evaluation of diseased area tracking by template matching for kidney cross-sectional image A. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, embodiments of the ultrasonic diagnostic treatment system, ultrasonic diagnostic treatment method, program, and affected area tracking evaluation method of the present invention will be described.

[0030] First Embodiment FIG. 1 is a diagram showing an example of an ultrasonic diagnostic treatment system 1 according to the first embodiment. In the example shown in FIG. 1, an ultrasound treatment and diagnosis system 1 of the first embodiment is used for treatment and / or diagnosis using ultrasound images. The ultrasound treatment diagnosis system 1 includes an image acquisition unit 11, an image group storage unit 12, an organ mask generation unit 13, an acoustic shadow mask generation unit 14, a complement target image setting unit 15, a reference image search unit 16, a complement area extraction unit 17, a composite complement image generation unit 18, and a composite complement image display unit 19. The image acquiring unit 11 acquires an ultrasound image. The image acquiring unit 11 acquires an ultrasound image obtained using an ultrasound probe or the like, for example, similarly to the technology described in Patent Document 1. The ultrasound image acquired by the image acquiring unit 11 may be an ultrasound image obtained using an ultrasound probe corresponding to a two-dimensional image display, or an ultrasound image obtained using a three-dimensional probe corresponding to a three-dimensional image display.

[0031] Fig. 2 is a diagram showing an example of an ultrasound image UM acquired by the image acquisition unit 11. In detail, Fig. 2(A) shows an example of an ultrasound image UM acquired by the image acquisition unit 11, and Fig. 2(B) shows an example of a composite complement image CM generated by the ultrasound treatment diagnosis system 1 of the first embodiment. In the example shown in FIG. 2, the organ included in the ultrasound image UM acquired by the image acquisition unit 11 is a kidney, but in other examples, the organ included in the ultrasound image UM acquired by the image acquisition unit 11 may be an organ other than a kidney. As shown in Fig. 2(A), an acoustic shadow may be included in the ultrasound image UM acquired by the image acquisition unit 11. In such a case, in the ultrasound treatment diagnosis system 1 of the first embodiment, as described later, a process of complementing the acoustic shadow shown in Fig. 2(A) is performed, and a composite complemented image CM as shown in Fig. 2(B) is generated. In the example shown in FIG. 2, the organ tissue in the defect area caused by the acoustic shadow shown in FIG. 2(A) on the same cross-sectional video of the kidney is reproduced as shown in FIG. 2(B) by the ultrasound treatment and diagnosis system 1 of the first embodiment.

[0032] In the example shown in FIG. 1, the image group storage unit 12 stores a plurality of ultrasound images acquired by the image acquisition unit 11. FIG. 3 is a diagram showing an example of a process in which ultrasound images UMk, UMk-1, ..., UMk-i, ..., UM0 obtained using an ultrasound probe are acquired by the image acquisition unit 11 and stored in the image group storage unit 12. In the example shown in FIG. 3, ultrasound image UM0 with identification number "0" indicating the newness (oldness) of the ultrasound image obtained using the ultrasound probe (i.e., in the example shown in FIG. 3, ultrasound image UM0 obtained first using the ultrasound probe) is stored in the image group storage unit 12. Furthermore, ultrasound images with identification numbers from "0" to "k-1" are stored in the image group storage unit 12. Furthermore, ultrasound image UMk with identification number "k" (i.e., in the example shown in FIG. 3, the latest ultrasound image UMk obtained using the ultrasound probe) is about to be stored in the image group storage unit 12.

[0033] In the example shown in FIG. 1, the organ mask generating unit 13 generates an organ mask corresponding to an organ included in an ultrasound image acquired by the image acquiring unit 11. In the example shown in Figure 3, the organ mask generation unit 13 generates organ mask GM0 corresponding to the organ contained in ultrasound image UM0, organ mask GMk-i corresponding to the organ contained in ultrasound image UMk-i, organ mask GMk-1 corresponding to the organ contained in ultrasound image UMk-1, etc., and these are stored in the image group storage unit 12 together with ultrasound images UM0, ..., UMk-i, ..., UMk-1, etc. Also, in the example shown in Figure 3, the organ mask generation unit 13 generates an organ mask GMk corresponding to the organ contained in the ultrasound image UMk, and the organ mask GMk is to be stored in the image group storage unit 12 together with the ultrasound image UMk and the like.

[0034] In the example shown in Figure 1, the organ mask generation unit 13 generates an organ mask by, for example, tracing (handwriting) the outline of an organ contained in an ultrasound image acquired by the image acquisition unit 11 by a user of the ultrasound treatment diagnosis system 1 of the first embodiment. In another example, the organ mask generating unit 13 may have a learning function and automatically generate an organ mask corresponding to the organ included in an ultrasound image acquired by the image acquiring unit 11, based on the ultrasound image. In this example, a pair of an ultrasound image including an organ and an organ mask corresponding to the organ included in the ultrasound image (i.e., an organ mask to be generated by the organ mask generating unit 13) is provided to the organ mask generating unit 13 as training data, and the organ mask generating unit 13 is trained. In this example, the organ mask generating unit 13 automatically generates an organ mask by performing organ segmentation using U-Net described in the following document. Olaf Ronneberger, et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, Medical Image Computing and Computer-Assisted Intervention - MICCAI, 2015, pp 234-241. As described in "U-Net: Semantic Segmentation by Deep Learning (https: / / blog.negativemind.com / 2019 / 03 / 15 / semantic-segmentation-by-u-net / )", U-Net is one of the segmentation (subregion extraction) methods for medical images proposed in U-Net: Convolutional Networks for Biomedical Image Segmentation, presented at MICCAI (Medical Image Computing and Computer-Assisted Intervention) 2015. Since U-Net does not have a fully connected network and is composed of convolutional layers, it is not necessary to fix the input image size. U-Net has an almost symmetrical Encoder-Decoder structure, and the feature map downsampled through Encoder pooling is upsampled by the Decoder. U-Net introduces an approach called "skip connection" that concatenates the feature maps output at each layer of the Encoder to the feature maps of the corresponding layers of the Decoder.

[0035] FIG. 4 is a diagram showing a comparison between an example in which an organ mask GMA is generated by the organ mask generating unit 13 using handwritten data and an example in which an organ mask GMB is automatically generated by the organ mask generating unit 13. In the example shown in the lower left of Figure 4, the organ mask generation unit 13 generates an organ mask GMA by using a traced (handwritten) outline of the organ contained in the ultrasound image UM acquired by the image acquisition unit 11. In the example shown in the lower right of Figure 4, an organ mask generation unit 13 with learning function performs organ segmentation using U-Net to automatically generate an organ mask GMB corresponding to the organ contained in the ultrasound image UM acquired by the image acquisition unit 11 from the ultrasound image UM.

[0036] In the example shown in FIG. 1, the acoustic shadow mask generating unit 14 generates an acoustic shadow mask corresponding to an acoustic shadow included in an ultrasound image acquired by the image acquiring unit 11. In the example shown in FIG. 3, the acoustic shadow mask generation unit 14 generates an acoustic shadow mask AM0 corresponding to the acoustic shadow AS included in the ultrasound image UM0, an acoustic shadow mask AMk-i corresponding to the acoustic shadow AS included in the ultrasound image UMk-i, an acoustic shadow mask AMk-1 corresponding to the acoustic shadow AS included in the ultrasound image UMk-1, etc., which are stored in the image group storage unit 12 together with the ultrasound images UM0, ..., UMk-i, ..., UMk-1, etc. In the example shown in FIG. 3, the acoustic shadow mask generation unit 14 generates an acoustic shadow mask AMk corresponding to the acoustic shadow AS contained in the ultrasound image UMk, and the acoustic shadow mask AMk is to be stored in the image group storage unit 12 together with the ultrasound image UMk and the like.

[0037] In the example shown in FIG. 1, the acoustic shadow mask generating unit 14 generates an acoustic shadow mask by, for example, tracing (handwriting) the contours of acoustic shadows contained in an ultrasound image acquired by the image acquiring unit 11 by a user of the ultrasound treatment diagnosis system 1 of the first embodiment. In another example, the acoustic shadow mask generating unit 14 may have a learning function and automatically generate an acoustic shadow mask corresponding to an acoustic shadow included in an ultrasound image acquired by the image acquiring unit 11, based on the ultrasound image. In this example, a pair of an ultrasound image including an acoustic shadow and an acoustic shadow mask corresponding to the acoustic shadow included in the ultrasound image (i.e., an acoustic shadow mask to be generated by the acoustic shadow mask generating unit 14) is provided to the acoustic shadow mask generating unit 14 as training data, and the acoustic shadow mask generating unit 14 is trained. In this example, the acoustic shadow mask generating unit 14 automatically generates an acoustic shadow mask by performing acoustic shadow segmentation by U-Net described in the above-mentioned literature.

[0038] In the example shown in FIG. 1, the complement target image setting unit 15 sets a complement target image which is an ultrasound image acquired by the image acquisition unit 11, contains an acoustic shadow, and needs to be complemented. In the example shown in Figure 3, the complement target image setting unit 15 sets the ultrasound image UMk acquired by the image acquisition unit 11 (i.e., in the example shown in Figure 3, the latest ultrasound image UMk to be stored in the image group storage unit 12) as the complement target image.

[0039] 1, the reference image search unit 16 searches for a reference image, which is an ultrasound image suitable for complementing an acoustic shadow included in an image to be complemented, from among a plurality of ultrasound images stored by the image group storage unit 12. In detail, the reference image search unit 16 searches for a reference image based on an organ mask corresponding to an organ included in the image to be complemented, an acoustic shadow mask corresponding to an acoustic shadow included in the image to be complemented, an organ mask corresponding to an organ included in a reference candidate image, which is each of a plurality of ultrasound images stored by the image group storage unit 12, and an acoustic shadow mask corresponding to an acoustic shadow included in the reference candidate image. 3, the reference image search unit 16 searches for a reference image RM that is an ultrasound image suitable for complementing an acoustic shadow AS included in an ultrasound image UMk as an image to be complemented from a plurality of ultrasound images UMk-1, ..., UMk-i, ..., UM0 stored by the image group storage unit 12. In detail, the reference image search unit 16 searches for a reference image RM based on an organ mask GMk corresponding to an organ included in the ultrasound image UMk as an image to be complemented, an acoustic shadow mask AMk corresponding to an acoustic shadow AS included in the ultrasound image UMk as an image to be complemented, organ masks GMk-1, ..., GMk-i, ..., GM0 corresponding to organs included in reference candidate images, which are each a plurality of ultrasound images UMk-1, ..., UMk-i, ..., UM0 stored by the image group storage unit 12, and acoustic shadow masks AMk-1, ..., AMk-i, ..., AM0 corresponding to acoustic shadow AS included in the reference candidate images (ultrasound images UMk-1, ..., UMk-i, ..., UM0).

[0040] In the example shown in FIG. 1, the complement region extraction unit 17 extracts, from the reference image searched for by the reference image search unit 16, a complement region that is used to complement an acoustic shadow included in the complement target image. In the example shown in FIG. 3, the complement area extraction unit 17 extracts, from the reference image RM, a complement area CA used for complementing an acoustic shadow AS included in an ultrasound image UMk as an image to be complemented.

[0041] In the example shown in FIG. 1, the synthetic complement image generating unit 18 synthesizes and complements the acoustic shadow included in the image to be complemented with the complement area extracted by the complement area extracting unit 17. In the example shown in Fig. 3, the synthetic complement image generating unit 18 synthetically complements the acoustic shadow AS included in the ultrasound image UMk as the image to be complemented by using the complement area CA included in the reference image RM. That is, the synthetic complement image generating unit 18 replaces the acoustic shadow AS included in the ultrasound image UMk as the image to be complemented by the complement area CA included in the reference image RM to generate a synthetic complement image CM (see Fig. 2(B) and Fig. 3) from the ultrasound image UMk as the image to be complemented.

[0042] In the example shown in FIG. 1, the composite complement image display unit 19 displays the composite complement image generated by the composite complement image generation unit 18. In the example shown in FIG. 3, the composite complemented image display unit 19 displays the composite complemented image CM so that the complemented area CA included in the composite complemented image CM and the area other than the complemented area CA can be distinguished. In another example, as shown in FIG. 2(B), the composite complement image display unit 19 may display the composite complement image CM so that the complement area CA contained in the composite complement image CM cannot be distinguished from the area other than the complement area CA (i.e., so that the boundary between the complement area CA and the area other than the complement area CA cannot be seen).

[0043] Fig. 5 is a diagram for explaining an organ mask generation experiment conducted by the present inventors. In detail, Fig. 5(A) shows an example of an ultrasound image for learning used in the organ mask generation experiment, and Fig. 5(B) shows an example of an organ mask for learning used in the organ mask generation experiment. That is, in the organ mask generation experiment, a pair of the ultrasound image shown in Fig. 5(A) and the organ mask shown in Fig. 5(B) was used as training data for learning. Fig. 5(C) shows an example of an ultrasound image for verification used in the organ mask generation experiment, and Fig. 5(D) shows an example of an organ mask for verification used in the organ mask generation experiment. In the organ mask generation experiment shown in Figure 5, an ultrasound diagnostic phantom model with an upper abdominal lesion described in the following literature was used to obtain an ultrasound image. https: / / www.kyotokagaku.com / jp / educational / products / detail03 / pdf / us-1b_catalog.pdf

[0044] In the organ mask generation experiment shown in Fig. 5, kidney ultrasound images captured from various angles were used. To expand the data of the captured kidney ultrasound images, the captured kidney ultrasound images were rotated, slid vertically and horizontally (translated), sheared, and scaled. The captured kidney ultrasound images (i.e., ultrasound images that do not contain acoustic shadows) were randomly assigned fan-shaped acoustic shadows. In addition, in the organ mask generation experiment shown in Figure 5, 1080 kidney organ cross-sectional images were used as learning data, 270 kidney organ cross-sectional images were used as verification data, and 130 kidney organ cross-sectional images were used as evaluation data.

[0045] 1, the reference image search unit 16 searches for a reference image based on the similarity between an organ mask corresponding to an organ included in the image to be complemented and an organ mask corresponding to an organ included in the reference candidate image. In detail, the reference image search unit 16 calculates the similarity between an organ mask corresponding to an organ included in the image to be complemented and an organ mask corresponding to an organ included in the reference candidate image by using the Dice coefficient.

[0046] FIG. 6 is a diagram for explaining the experimental conditions in the organ mask generation experiment conducted by the present inventors. In the organ mask generation experiments conducted by the inventors, the similarity between an organ mask automatically generated from an ultrasound image containing an organ by performing organ segmentation using U-Net (i.e., an estimated organ mask) and the organ mask to be generated (i.e., the correct organ mask) was calculated using the Dice coefficient. The upper left part of Figure 6 shows that the higher the similarity between the estimated organ mask and the ground truth, the closer the loss function is to 1. In the organ mask generation experiment conducted by the inventors, as shown in FIG. 6, the weights of the 600th epoch and the 476th epoch were used as the number of epochs. Adam (learning rate = 0.001) was used as the optimal function. The Dice coefficient was used as the loss function. The image preprocessing was performed as follows: 400 x 400 → grayscale → 256 x 256 → normalization (0 to 1) → acoustic shading. In an organ mask generation experiment conducted by the inventors, the relationship between the loss function (Loss) and the number of epochs (Epoch) shown in the right part of FIG. 6 was obtained under the execution environment (computer) shown in the right part of FIG. 6.

[0047] FIG. 7 is a diagram for explaining the results of an organ mask generation experiment conducted by the present inventors. In detail, Figure 7(A) shows, from the left, an ultrasound image including an organ in an example where the Dice coefficient is 0.976, an organ mask automatically generated from the ultrasound image (estimated organ mask), and an organ mask to be generated (correct organ mask). Figure 7(B) shows, from the left, an ultrasound image containing an organ in an example where the Dice coefficient was 0.974, an organ mask automatically generated from the ultrasound image (estimated organ mask), and the organ mask that should be generated (correct organ mask). Figure 7(C) shows, from the left, an ultrasound image containing an organ in an example where the Dice coefficient was 0.959, an organ mask automatically generated from the ultrasound image (estimated organ mask), and the organ mask that should be generated (correct organ mask). Figure 7(D) shows, from the left, an ultrasound image containing an organ in an example where the Dice coefficient was 0.951, an organ mask automatically generated from the ultrasound image (estimated organ mask), and the organ mask that should be generated (correct organ mask). In an organ mask generation experiment conducted by the present inventors, the average Dice coefficient was 0.970, as shown in Fig. 7. In other words, in an organ mask generation experiment conducted by the present inventors, organ regions could be extracted with high accuracy from ultrasound images containing acoustic shadows and organs (i.e., highly accurate organ masks could be automatically generated). By using a technique similar to that used in the organ mask generation experiments conducted by the inventors, it is possible to extract acoustic shadow regions with high accuracy from ultrasound images that contain acoustic shadows and organs (i.e., it is possible to automatically generate highly accurate acoustic shadow masks).

[0048] In the example shown in Fig. 1, the reference image search unit 16 searches for a reference image based on the similarity between an acoustic shadow mask corresponding to an acoustic shadow included in the image to be complemented and an acoustic shadow mask corresponding to an acoustic shadow included in the reference candidate image. In detail, the reference image search unit 16 calculates the similarity between an acoustic shadow mask corresponding to an acoustic shadow included in the image to be complemented and an acoustic shadow mask corresponding to an acoustic shadow included in the reference candidate image by using the Dice coefficient.

[0049] In addition, in the example shown in Figure 1, the reference image search unit 16 aligns the organ mask corresponding to the organ included in the reference candidate image and the acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image with the organ mask corresponding to the organ included in the image to be complemented and the acoustic shadow mask corresponding to the acoustic shadow included in the image to be complemented, so that the contours of the organs in the organ mask corresponding to the organ included in the image to be complemented and the organ mask corresponding to the organ included in the reference candidate image match.

[0050] FIG. 8 is a diagram for explaining an example of positioning by the reference image searching unit 16. In FIG. In the example shown in Figure 8, the reference image search unit 16 translates (in detail, translates, for example, toward the center of gravity) the organ mask corresponding to the organ included in the reference candidate image and the acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image in order to match the contours of the organs in the organ mask corresponding to the organ included in the image to be complemented and the organ mask corresponding to the organ included in the reference candidate image. Furthermore, in the example shown in Figure 8, the reference image search unit 16 rotates (in detail, for example, rotates about the center of gravity) the organ mask corresponding to the organ included in the reference candidate image and the acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image in order to match the contours of the organs in the organ mask corresponding to the organ included in the image to be complemented and the organ mask corresponding to the organ included in the reference candidate image. In another example, the reference image search unit 16 may enlarge or reduce the organ mask corresponding to the organ included in the reference candidate image and the acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image in order to match the contours of the organs in the organ mask corresponding to the organ included in the image to be complemented and the organ mask corresponding to the organ included in the reference candidate image.

[0051] In the example shown in Figure 1, the reference image search unit 16 selects the reference candidate image as the reference image when the contours of the organ in the organ mask corresponding to the organ included in the image to be complemented match those in the organ mask corresponding to the organ included in the reference candidate image, and the contours of the acoustic shadow in the acoustic shadow mask corresponding to the acoustic shadow included in the image to be complemented do not overlap with those in the acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image (for example, the upper right part of Figure 8). On the other hand, when the contours of an organ match between the organ mask corresponding to the organ included in the image to be complemented and the organ mask corresponding to the organ included in the reference candidate image, and the contours of the acoustic shadow at least partially overlap between the acoustic shadow mask corresponding to the acoustic shadow included in the image to be complemented and the acoustic shadow mask corresponding to the acoustic shadow included in the reference candidate image (for example, the case of the lower right part of Figure 8), the reference image search unit 16 searches for a reference image by using other ultrasound images among the multiple ultrasound images stored by the image group storage unit 12 as reference candidate images.

[0052] In the example shown in Figure 1, the reference image search unit 16 evaluates whether the organ contours of an organ mask corresponding to an organ included in the image to be complemented match those of an organ mask corresponding to an organ included in the reference candidate image by using the Dice coefficient, as shown in Figure 8.

[0053] FIG. 9 is a diagram for explaining an example of extraction of a complement area CA from a reference image RM by the complement area extraction unit 17 and synthesis and complementation of the complement target image by the composite complement image generation unit 18 (generation of a composite complement image CM). 9, the complement area extraction unit 17 extracts a complement area CA used to complement an acoustic shadow AS included in an ultrasound image as a complement target image from a reference image RM. The synthetic complement image generation unit 18 synthesizes and complements the acoustic shadow AS included in the complement target image using the complement area CA included in the reference image RM. In other words, the synthetic complement image generation unit 18 replaces the acoustic shadow AS included in the complement target image with the complement area CA included in the reference image RM to generate a synthetic complement image CM from the complement target image. In the example shown in FIG. 9, the shape of the complement area CA extracted from the reference image RM by the complement area extraction unit 17 is the same as the shape of the acoustic shadow AS included in the image to be complemented. In another example, the shape of the complement area CA extracted from the reference image RM by the complement area extraction unit 17 may be different from the shape of the acoustic shadow AS included in the image to be complemented. In this example, the complement area CA is smaller than the acoustic shadow AS included in the image to be complemented, and the complement area CA includes only the area corresponding to the part inside the outline of the organ. In this example, the area of ​​the acoustic shadow AS included in the image to be complemented that corresponds to the part outside the outline of the organ is left as it is in the composite complemented image CM.

[0054] FIG. 10 is a diagram for explaining a synthesis and complementation experiment conducted by the present inventors. In the organ mask generation experiment shown in Figure 10, the ultrasound diagnostic phantom model with an upper abdominal lesion described in the above-mentioned literature was used to obtain ultrasound images, and ultrasound images were obtained while moving the model in the direction of respiratory motion.

[0055] In the organ mask generation experiment shown in Figure 10, ultrasound videos of a kidney phantom of the same cross section were used. Respiratory movements were reproduced by a bed-type robot. One type of ultrasound image, kidney cross section image A, taken from the abdomen, and one type of kidney cross section image B, taken from the back, were used. Pre-annotated data were used as the organ mask. Fan-shaped acoustic shadows were randomly created. 317 kidney cross section images A taken from the abdomen and 280 kidney cross section images B taken from the back were used.

[0056] FIG. 11 is a diagram for explaining the evaluation method of the synthesis and complementation experiment conducted by the present inventors. In the evaluation method of the synthesis completion experiment shown in Fig. 11, the similarity between the image to be complemented and the synthesis completion image was evaluated. In detail, the similarity between the image to be complemented and the correct image of the organ area GA in the complement area CA was evaluated. In the similarity evaluation shown in Fig. 11, zero-mean normalized cross-correlation (ZNCC) was used. The zero-mean normalized cross-correlation has a value between -1 and 1, and the closer the value of the zero-mean normalized cross-correlation is to 1, the higher the similarity of the complementation area CA to the ground truth image of the organ area GA. The value of the zero-mean normalized cross-correlation as the target similarity was set to 0.700.

[0057] In a synthesis and complementation experiment conducted by the present inventors, the average ZNCC of the kidney region was 0.740 when 286 of the above-mentioned kidney cross-sectional images A were used to evaluate the similarity of the complemented kidney region. The average ZNCC of the kidney region was 0.630 when 253 of the above-mentioned kidney cross-sectional images B were used. The average ZNCC of the total kidney region was 0.667.

[0058] Fig. 12 shows the results of a synthesis and completion experiment (image synthesis experiment) conducted by the present inventors on kidney cross-sectional image A shown in Fig. 10. In detail, Fig. 12 shows a similarity comparison of kidney completion regions using ZNCC. The horizontal axis of Fig. 12 shows the image number, and the vertical axis of Fig. 12 shows the similarity. In the similarity comparison of kidney completion regions using ZNCC, the average ZNCC was 0.740. In a synthesis completion experiment conducted by the inventors, it was possible to generate a synthesis completion image that achieved the target similarity of 0.700 for the kidney region. In other words, the contours and internal tissues were smoothly synthesized in the synthesis completion image.

[0059] FIG. 13 is a flowchart showing an example of the process executed in the ultrasound diagnostic treatment system 1 of the first embodiment. In the example shown in FIG. 13, in step S11, the image acquiring unit 11 acquires an ultrasound image. Next, in step S12, the image group storage unit 12 stores the multiple ultrasound images acquired in step S11. Next, in step S13, the organ mask generating unit 13 generates an organ mask corresponding to the organ included in the ultrasound image acquired in step S11. In addition, in step S14, the acoustic shadow mask generator 14 generates an acoustic shadow mask corresponding to the acoustic shadow included in the ultrasound image acquired in step S11. In step S15, the complement target image setting unit 15 sets a complement target image which is an ultrasound image acquired in step S11, includes an acoustic shadow, and needs to be complemented. Next, in step S16, the reference image search unit 16 searches for a reference image, which is an ultrasound image suitable for complementing the acoustic shadow contained in the complement target image set in step S15, from the multiple ultrasound images stored in step S12. Next, in step S17, the complement region extraction unit 17 extracts a complement region, which is a region used to complement the acoustic shadow included in the complement target image set in step S15, from the reference image searched for in step S16. Next, in step S18, the composite complement image generating unit 18 composites and complements the acoustic shadow included in the complement target image set in step S15 with the complement area extracted in step S17. Next, in step S19, the composite complement image display unit 19 displays the composite complement image generated in step S18.

[0060] As described above, the ultrasound treatment and diagnosis system 1 of the first embodiment can complement the acoustic shadow contained in the ultrasound image of the organ without the need to estimate the contour of the organ.

[0061] <Second embodiment> A second embodiment of the ultrasonic diagnostic treatment system, ultrasonic diagnostic treatment method, program, and affected area tracking evaluation method of the present invention will be described below. The ultrasonic diagnostic treatment system 1 of the second embodiment is configured similarly to the ultrasonic diagnostic treatment system 1 of the first embodiment, except for the points described below. Therefore, the ultrasonic diagnostic treatment system 1 of the second embodiment can achieve the same effects as the ultrasonic diagnostic treatment system 1 of the first embodiment, except for the points described below.

[0062] As described above, in the ultrasound diagnostic treatment system 1 of the first embodiment, the composite complemented image generated by the composite complemented image generating unit 18 is not evaluated using template matching. On the other hand, in the ultrasound treatment diagnosis system 1 of the second embodiment, as described later, in addition to the similarity evaluation shown in FIG. 11, a tracking evaluation of the affected area included in the composite complement image is performed on the composite complement image generated by the composite complement image generation unit 18 by using template matching.

[0063] FIG. 14 is a diagram for explaining a method for evaluating the tracking of an affected area included in a composite-complemented image in a composite-complemented experiment conducted by the present inventors. 14, the evaluation method of the synthetic complement experiment involves tracking of the affected area by template matching in order to evaluate the visibility of the synthetic complement image generated by the synthetic complement image generating unit 18. For tracking (tracking) by template matching, a technique similar to that described in, for example, JP2019-033960A is used. In the evaluation method of the synthesis and completion experiment shown in Fig. 14, the template was scanned in the correct image, the image to be completed, and the synthesis and completion image, and the position with the highest ZNCC was detected and set as the center of gravity coordinate (x, y). In addition, the target accuracy of the Euclidean distance [mm] was set to within 1 mm. In Figure 14, "ED 正-影 " indicates the Euclidean distance between the correct image and the image to be complemented, and "ED 正-合 " indicates the Euclidean distance between the correct image and the synthesized and complemented image.

[0064] In a synthesis completion experiment conducted by the inventors, when the above-mentioned kidney cross-sectional image A (moving image A) was used to evaluate the tracking of the affected area by template matching, the average Euclidean distance between the correct image and the acoustic shadowed image (image to be completed) was 35.2 [mm], and the average Euclidean distance between the correct image and the synthetic completion image was 0.53 [mm]. The average ZNCC of the affected area of ​​the correct image was 0.786, the average ZNCC of the affected area of ​​the acoustic shadowed image (image to be completed) was 0.635, and the average ZNCC of the affected area of ​​the synthetic completion image was 0.761. When the above-mentioned kidney cross-sectional image B (video image B) was used, the average Euclidean distance between the correct image and the acoustic shaded image (image to be complemented) was 17.3 [mm], and the average Euclidean distance between the correct image and the synthetic complemented image was 0.33 [mm]. The average ZNCC of the affected area of ​​the correct image was 0.833, the average ZNCC of the affected area of ​​the acoustic shaded image (image to be complemented) was 0.597, and the average ZNCC of the affected area of ​​the synthetic complemented image was 0.788. Combining the results for kidney cross-sectional image A (video A) and kidney cross-sectional image B (video B), the average Euclidean distance between the correct image and the acoustic shaded image (image to be complemented) was 25.7 [mm], and the average Euclidean distance between the correct image and the synthetic complemented image was 0.42 [mm]. The average ZNCC of the affected area of ​​the correct image was 0.811, the average ZNCC of the affected area of ​​the acoustic shaded image (image to be complemented) was 0.615, and the average ZNCC of the affected area of ​​the synthetic complemented image was 0.775.

[0065] FIG. 15 is a diagram showing the results of a synthesis and complementation experiment (image synthesis experiment) on a kidney cross-sectional image A (moving image A) performed by the present inventors using the ultrasound diagnostic and treatment system 1 of the second embodiment. In detail, Fig. 15(A) shows the Euclidean distance between a correct image (correct video) and a synthetic complement image, and the Euclidean distance between a correct image (correct video) and a complement target image. The horizontal axis of Fig. 15(A) shows the image number, and the vertical axis of Fig. 15(A) shows the Euclidean distance between a correct video (correct video) and a synthetic complement image generated by the synthetic complement image generating unit 18, and the Euclidean distance between a correct video (correct video) and a complement target image set by the complement target image setting unit 15. As described above, the average Euclidean distance between a correct video (correct video) and a synthetic complement image is 0.53 mm, and the average Euclidean distance between a correct video (correct video) and a synthetic complement image is 35.2 mm. Figure 15(B) shows a comparison of the similarity of the detected regions. The horizontal axis of Figure 15(B) shows the image number, and the vertical axis of Figure 15(B) shows the similarity of the correct image, the complement image (acoustic shadowed image), and the synthetic complement image. As mentioned above, the average ZNCC of the affected area of ​​the correct image was 0.786, and the average ZNCC of the affected area of ​​the synthetic complement image was 0.761.

[0066] FIG. 16 is a diagram for explaining evaluation of diseased area tracking by template matching for kidney cross-sectional image A. In detail, Fig. 16(A) shows a state where evaluation of diseased part tracking by template matching is performed on a complement target image (complement target moving image) including an acoustic shadow. In the evaluation shown in Fig. 16(A), since the complement target image includes an acoustic shadow, the result is that tracking of the diseased part (kidney) is not possible. Fig. 16(B) shows the state where the evaluation of the tracking of the affected area by template matching is performed on the correct image (correct moving image). In the evaluation shown in Fig. 16(B), since the correct image does not contain an acoustic shadow, the system was able to track the affected area (kidney). Fig. 16(C) shows the state where the evaluation of the tracking of the affected area by template matching is performed on the synthetic complement image (synthetic complement video image). In the evaluation shown in Fig. 16(C), the acoustic shadow is complemented by the complement area (i.e., the acoustic shadow is not included in the synthetic complement image), so the affected area (kidney) could be tracked.

[0067] In a synthesis and complementation experiment conducted by the inventors, a synthesis and complementation image was generated that achieved a diseased area tracking accuracy of 0.42 [mm]. In other words, sufficient accuracy was achieved for monitoring the treated diseased area. It was also confirmed that when the similarity of the synthesis and complementation area is low, deviations in the contour and internal tissue occur, and these deviations lead to reduced visibility.

[0068] The ultrasound diagnostic and treatment system 1 of the second embodiment can improve the tracking of organs and enable continuous diagnosis in the presence of acoustic shadows.

[0069] Although the present invention has been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions can be made without departing from the scope of the present invention. The configurations described in the above-mentioned embodiments and examples may be combined.

[0070] In addition, the whole or part of the functions of each unit of the ultrasound diagnostic treatment system 1 in the above-mentioned embodiment may be realized by recording a program for realizing these functions in a computer-readable recording medium, reading the program recorded in the recording medium into a computer system, and executing it. Note that the "computer system" here includes hardware such as an OS and peripheral devices. In addition, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage units such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" may also include those that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system. [Explanation of symbols]

[0071] 1...ultrasound treatment and diagnosis system, 11...image acquisition unit, 12...image group storage unit, 13...organ mask generation unit, 14...acoustic shadow mask generation unit, 15...complementation target image setting unit, 16...reference image search unit, 17...complementation region extraction unit, 18...synthetic complemented image generation unit, 19...synthetic complemented image display unit

Claims

1. An ultrasound diagnostic and treatment system for use in treatment and / or diagnosis using ultrasound images, comprising: an image acquisition unit for acquiring an ultrasound image; an image group storage unit that stores a plurality of ultrasound images acquired by the image acquisition unit; an organ mask generating unit that generates an organ mask corresponding to an organ included in the ultrasound image acquired by the image acquiring unit; an acoustic shadow mask generating unit that generates an acoustic shadow mask corresponding to an acoustic shadow included in the ultrasound image acquired by the image acquiring unit; a complementation target image setting unit that sets a complementation target image, which is an ultrasound image acquired by the image acquisition unit, that includes an acoustic shadow and needs to be complemented; a reference image search unit that searches for a reference image that is an ultrasound image suitable for complementing an acoustic shadow included in the complement target image from among the plurality of ultrasound images stored by the image group storage unit; a complementation region extraction unit that extracts, from the reference image, a complementation region that is used to complement an acoustic shadow included in the complement target image; a synthetic complement image generating unit that synthesizes and complements the acoustic shadow included in the complement target image with the complementary area extracted by the complementary area extracting unit, the reference image search unit searches for the reference image based on an organ mask corresponding to an organ included in the image to be complemented, an acoustic shadow mask corresponding to an acoustic shadow included in the image to be complemented, an organ mask corresponding to an organ included in a reference candidate image, which is each of a plurality of ultrasound images stored by the image group storage unit, and an acoustic shadow mask corresponding to an acoustic shadow included in the reference candidate image; Ultrasound treatment and diagnostic system.

2. The organ mask generation unit generates an organ mask by performing organ segmentation using a segmentation method for medical images. The ultrasound diagnostic and treatment system according to claim 1 .

3. The acoustic shadow mask generating unit generates an acoustic shadow mask by performing acoustic shadow segmentation using a segmentation method for medical images. The ultrasound diagnostic and treatment system according to claim 1 .

4. the reference image search unit searches for the reference image based on a similarity between an organ mask corresponding to an organ included in the complement target image and an organ mask corresponding to an organ included in the reference candidate image; The ultrasound diagnostic and treatment system according to claim 1 .

5. The reference image search unit calculates a similarity between an organ mask corresponding to an organ included in the complement target image and an organ mask corresponding to an organ included in the reference candidate image by using a Dice coefficient. The ultrasonic diagnostic and treatment system according to claim 4.

6. The reference image search unit searches for the reference image based on a similarity between an acoustic shadow mask corresponding to an acoustic shadow included in the complement target image and an acoustic shadow mask corresponding to an acoustic shadow included in the reference candidate image. The ultrasound diagnostic and treatment system according to claim 1 .

7. The reference image search unit calculates a similarity between an acoustic shadow mask corresponding to an acoustic shadow included in the complement target image and an acoustic shadow mask corresponding to an acoustic shadow included in the reference candidate image by using a Dice coefficient. The ultrasonic diagnostic and treatment system according to claim 6.

8. The reference image search unit includes: aligning an organ mask corresponding to an organ included in the reference candidate image and an acoustic shadow mask corresponding to an acoustic shadow included in the reference candidate image with respect to an organ mask corresponding to an organ included in the image to be complemented and an acoustic shadow mask corresponding to an acoustic shadow included in the image to be complemented, so that the contours of the organs in the organ mask corresponding to the organ included in the image to be complemented and the organ mask corresponding to the organ included in the reference candidate image coincide with each other; When an organ mask corresponding to an organ included in the image to be complemented and an organ mask corresponding to an organ included in the reference candidate image match in outline, and an acoustic shadow mask corresponding to an acoustic shadow included in the image to be complemented and an acoustic shadow mask corresponding to an acoustic shadow included in the reference candidate image do not overlap in outline, the reference candidate image is selected as the reference image. The ultrasound diagnostic and treatment system according to claim 1 .

9. The reference image search unit includes: when an organ mask corresponding to an organ included in the image to be complemented and an organ mask corresponding to an organ included in the reference candidate image have identical contours, and when an acoustic shadow mask corresponding to an acoustic shadow included in the image to be complemented and an acoustic shadow mask corresponding to an acoustic shadow included in the reference candidate image at least partially overlap with each other, another ultrasound image among the plurality of ultrasound images stored by the image group storage unit is set as the reference candidate image, and the reference image is searched for. The ultrasound diagnostic and treatment system according to claim 8.

10. The reference image search unit performing at least one of translation, rotation, enlargement, and reduction of an organ mask corresponding to an organ included in the reference candidate image and an acoustic shadow mask corresponding to an acoustic shadow included in the reference candidate image, in order to match the contours of the organs between the organ mask corresponding to the organ included in the image to be complemented and the organ mask corresponding to the organ included in the reference candidate image; The ultrasound diagnostic and treatment system according to claim 8.

11. The reference image search unit using a Dice coefficient to evaluate whether or not an organ contour matches an organ mask corresponding to an organ included in the image to be complemented and an organ mask corresponding to an organ included in the reference candidate image; The ultrasound diagnostic and treatment system according to claim 8.

12. a shape of the complement area extracted from the reference image by the complement area extraction unit is identical to a shape of an acoustic shadow included in the complement target image; The ultrasound diagnostic and treatment system according to claim 1 .

13. the shape of the complement area extracted from the reference image by the complement area extraction unit is different from the shape of the acoustic shadow included in the complement target image, the complemented region is smaller than an acoustic shadow included in the complement target image; The complementary region includes only a region corresponding to a portion inside the outline of an organ. The ultrasound diagnostic and treatment system according to claim 1 .

14. a composite complement image display unit that displays the composite complement image generated by the composite complement image generation unit, the composite complement image display unit displays the complement area included in the composite complement image and an area other than the complement area in a distinguishable manner; The ultrasound diagnostic and treatment system according to claim 1 .

15. The ultrasound images acquired by the image acquisition unit include ultrasound images acquired using a three-dimensional probe. The ultrasound diagnostic and treatment system according to claim 1 .

16. On the computer, An image acquisition step of acquiring an ultrasound image; an image group storage step of storing the plurality of ultrasound images acquired in the image acquisition step; an organ mask generating step of generating an organ mask corresponding to an organ included in the ultrasound image acquired in the image acquiring step; an acoustic shadow mask generating step of generating an acoustic shadow mask corresponding to an acoustic shadow included in the ultrasound image acquired in the image acquiring step; a complementation target image setting step of setting a complementation target image which is an ultrasound image acquired in the image acquisition step, the ultrasound image including an acoustic shadow and which needs to be complemented; a reference image searching step of searching for a reference image that is an ultrasound image suitable for complementing an acoustic shadow included in the complement target image from among the plurality of ultrasound images stored in the image group storing step; a complementation region extraction step of extracting, from the reference image, a complementation region used for complementing an acoustic shadow included in the complement target image; a synthetic complement image generating step of synthesizing and complementing an acoustic shadow included in the complement target image with the complement area extracted in the complement area extracting step, In the reference image search step, the reference image is searched for based on an organ mask corresponding to an organ included in the image to be complemented, an acoustic shadow mask corresponding to an acoustic shadow included in the image to be complemented, an organ mask corresponding to an organ included in a reference candidate image, which is each of the multiple ultrasound images stored in the image group storage step, and an acoustic shadow mask corresponding to an acoustic shadow included in the reference candidate image. program.

17. and performing a tracking evaluation of a diseased area included in the composite complement image by using template matching for the composite complement image generated by the composite complement image generating unit of the ultrasound diagnostic and treatment system according to claim 1. A method for assessing patient follow-up.

Images