Ultrasound diagnostic device, medical image processing device, medical image processing method, and program

Abstract

To search for an ultrasonic image for facilitating registration in a short time.SOLUTION: An ultrasonic diagnostic device includes a derivation unit, a control unit, and a registration unit. The derivation unit derives an image evaluation parameter defined beforehand for a plurality of two-dimensional ultrasonic images acquired by an ultrasonic probe. The control unit selects the two-dimensional ultrasonic image for registration from the plurality of two-dimensional ultrasonic images on the basis of an image evaluation parameter derivation result. The registration unit executes registration of a three-dimensional medical image and the two-dimensional ultrasonic image for registration.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The embodiments disclosed in this specification and the drawings relate to an ultrasound diagnostic apparatus, a medical image processing apparatus, a medical image processing method, and a program. [Background technology]

[0002] In the medical field, ultrasound diagnostic devices are used as medical imaging diagnostic devices that image the inside of a subject using ultrasound generated by multiple transducers (piezoelectric transducers) in an ultrasound probe. The ultrasound diagnostic device transmits ultrasound waves from an ultrasound probe connected to the ultrasound diagnostic device into the subject, generates echo signals based on the reflected waves, and obtains a desired ultrasound image through image processing.

[0003] For example, in order to determine the location and size of a lesion, it is necessary to grasp the spatial structure inside a subject. In this case, three-dimensional (3D) medical images obtained by a medical imaging diagnostic device other than the above-mentioned ultrasound diagnostic device are used. Examples of 3D medical images include 3D ultrasound images obtained by an ultrasound diagnostic device, as well as 3D images obtained by other medical imaging devices such as an X-ray CT (Computed Tomography) device or an MRI (Magnetic Resonance Imaging) device.

[0004] 3D medical images are generated in advance, for example, during the preoperative diagnostic stage. However, during surgery, issues such as the patient's position may cause discrepancies between the actual internal structure and the volume data in the 3D medical image, making it impossible to accurately depict the internal structure of the patient. Therefore, there is a need for real-time imaging of the surgical area, such as a lesion, using a non-invasive ultrasound diagnostic device during surgery. Compared to X-ray CT or MRI devices, ultrasound diagnostic devices can be used to perform registration as quickly as possible without limiting the location and patient condition, thereby reducing the burden on the patient during an abdominal opening, for example.

[0005] To perform registration as quickly as possible, it is important to select ultrasound images that facilitate registration as quickly as possible. For example, when an operator performs operations such as alignment by setting observable marks on both ultrasound images and 3D medical images, the operations are complex and time-consuming, placing a heavy burden on the subject in an open abdominal position. Therefore, performing registration using a single ultrasound image may result in a problem in which the registration results are not ideal. On the other hand, performing registration using multiple ultrasound images individually takes a long time to obtain ideal registration results. Therefore, it is desirable to be able to search for ultrasound images that facilitate registration as quickly as possible. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2017-202125 Summary of the Invention [Problem to be solved by the invention]

[0007] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to quickly search for ultrasound images that facilitate registration. However, the problems solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0008] The ultrasound diagnostic apparatus according to the present embodiment includes a derivation unit, a control unit, and a registration unit. The derivation unit derives predefined image evaluation parameters for a plurality of two-dimensional ultrasound images acquired by an ultrasound probe. The control unit selects a two-dimensional ultrasound image for registration from the plurality of two-dimensional ultrasound images based on the derivation results of the image evaluation parameters. The registration unit registers a three-dimensional medical image with the two-dimensional ultrasound image for registration. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an ultrasonic diagnostic apparatus according to this embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of a processing circuit of the ultrasonic diagnostic apparatus according to this embodiment. [Figure 3A] FIG. 3A is a diagram showing an example of a display of a 2D ultrasound image. [Figure 3B] FIG. 3B is a diagram showing an example of a displayed 2D ultrasound image. [Figure 3C] FIG. 3C is a diagram showing an example of a displayed 2D ultrasound image. [Figure 4] FIG. 4 is a diagram showing a display example of a 2D ultrasound image. [Figure 5A] FIG. 5A is a flowchart showing the processing of the ultrasonic diagnostic apparatus according to this embodiment. [Figure 5B] FIG. 5B is a flowchart showing the processing of the ultrasonic diagnostic apparatus according to this embodiment. [Figure 5C] FIG. 5C is a flowchart showing the processing of the ultrasonic diagnostic apparatus according to this embodiment. [Figure 6A] FIG. 6A shows a 2D ultrasound image including landmarks. [Figure 6B] FIG. 6B shows a 2D ultrasound image including landmarks. [Figure 7A] FIG. 7A shows 2D ultrasound images with different texture structures. [Figure 7B]FIG. 7B shows 2D ultrasound images with different texture structures. [Figure 8] FIG. 8 is a diagram for explaining a method for determining an orthogonal metric. [Figure 9A] FIG. 9A is a diagram for explaining a method for determining the coverage. [Figure 9B] FIG. 9B is a diagram for explaining a method for determining the coverage. [Figure 10] FIG. 10 is a diagram illustrating a specific example of registration using landmarks. [Figure 11] FIG. 11 is a flowchart showing a specific example of registration using landmarks. [Figure 12A] FIG. 12A shows a specific example of selecting multiple 2D ultrasound images as 2D ultrasound images for registration. [Figure 12B] FIG. 12B shows a specific example of selecting multiple 2D ultrasound images as 2D ultrasound images for registration. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an ultrasound diagnostic apparatus, a medical image processing apparatus, a medical image processing method, and a computer program for realizing the medical image processing method according to embodiments will be described with reference to the accompanying drawings. Note that the embodiments described below are merely examples and are not limited to the following embodiments. Furthermore, the content described in one embodiment can, in principle, be applied to other embodiments as well.

[0011] Fig. 1 is a block diagram showing an example of the configuration of an ultrasound diagnostic apparatus 1 according to this embodiment. As shown in Fig. 1, the ultrasound diagnostic apparatus 1 according to this embodiment includes an apparatus main body 100, an ultrasound probe 101, an input device 102, and a display 103. The ultrasound probe 101, the input device 102, and the display 103 are connected to the apparatus main body 100.

[0012] The ultrasonic probe 101 has a plurality of transducers (e.g., piezoelectric transducers), which generate ultrasonic waves based on drive signals supplied from a transmission / reception circuit 110 included in the device main body 100 (described later). The plurality of transducers included in the ultrasonic probe 101 also receive reflected waves from the subject P and convert them into electrical signals. The ultrasonic probe 101 also has a matching layer provided on the transducer, a backing material that prevents ultrasonic waves from propagating backward from the transducer, and the like. The ultrasonic probe 101 is also equipped with a magnetic sensor for acquiring position information of the ultrasonic probe 101.

[0013] The input device 102 includes an input device that can be operated by an operator and an input circuit that inputs signals from the input device. The input device can be realized by a tracking ball, a switch, a mouse, a keyboard, a touch panel that performs input operations by touching the operation surface, a touch screen that integrates a display screen and a touch panel, a non-contact input device that uses an optical sensor, a voice input device, etc. When the input device is operated by the operator, the input circuit generates a signal corresponding to the operation and outputs it to the processing circuit.

[0014] The display 103 is configured by a general display output device such as a liquid crystal display or an OLED (Organic Light Emitting Diode) display. The display 103 displays a GUI (Graphical User Interface) that allows the operator of the ultrasound diagnostic apparatus 1 to input various setting requests using the input device 102, and displays ultrasound image data generated in the apparatus main body 100. The display 103 is an example of a display unit.

[0015] The device main body 100 is a device that generates ultrasound image data based on reflected wave signals received by the ultrasound probe 101, and as shown in Fig. 1, has a transmission / reception circuit 110, a signal processing circuit 120, an image generation circuit 130, an image memory 140, a storage circuit 150, and a processing circuit 160. The transmission / reception circuit 110, the signal processing circuit 120, the image generation circuit 130, the image memory 140, the storage circuit 150, and the processing circuit 160 are connected to each other so that they can communicate with each other.

[0016] The transmission / reception circuit 110 controls the transmission of ultrasound waves by the ultrasound probe 101. For example, based on instructions from the processing circuit 160, the transmission / reception circuit 110 applies the above-mentioned drive signal (drive pulse) to the ultrasound probe 101 at a timing to which a predetermined transmission delay time is added for each transducer. As a result, the transmission / reception circuit 110 causes the ultrasound probe 101 to transmit an ultrasound beam, which is ultrasound waves focused into a beam shape. The transmission / reception circuit 110 also controls the reception of a reflected wave signal by the ultrasound probe 101. As described above, the reflected wave signal is a signal that is generated when ultrasound waves transmitted from the ultrasound probe 101 are reflected by the body tissue of the subject P. For example, based on instructions from the control circuit 170, the transmission / reception circuit 110 adds a predetermined delay time to the reflected wave signal received by the ultrasound probe 101 and performs an addition process. As a result, the reflected component from a direction corresponding to the reception directivity of the reflected wave signal is emphasized.

[0017] The signal processing circuit 120 performs various signal processing on the reflected wave signal received by the transmission / reception circuit 110. For example, the signal processing circuit 120 performs various signal processing on the reflected wave signal to generate data (B-mode data) in which the signal intensity at each sample point (observation point) is expressed as brightness. The signal processing circuit 120 also generates data (Doppler data) in which motion information based on the Doppler effect of a moving object is extracted at each sample point within the scanning region.

[0018] The image generation circuit 130 generates image data (ultrasound images) from data that has undergone various signal processes by the signal processing circuit 120, and performs various image processes on the ultrasound images. For example, the image generation circuit 130 generates a 2D ultrasound image in which the intensity of reflected waves is represented by brightness from two-dimensional (2D) B-mode data. The image generation circuit 130 also generates a 2D ultrasound image in which blood flow information is visualized from two-dimensional Doppler data.

[0019] Here, the image generation circuit 130 generates an ultrasound image for display by performing coordinate transformation according to the ultrasound scanning form of the ultrasound probe 101. For example, the B-mode data and Doppler data are ultrasound image data before scan conversion processing, and the data generated by the image processing circuit 140 is ultrasound image data for display after scan conversion processing. That is, the image generation circuit 130 generates 2D ultrasound image data for display from 2D ultrasound image data before scan conversion processing. Furthermore, the image generation circuit 130 generates a 3D ultrasound image by performing coordinate transformation on the three-dimensional (3D) B-mode data generated by the signal processing circuit 120. Furthermore, the image generation circuit 130 generates a 3D ultrasound image by performing coordinate transformation on the 3D Doppler data generated by the signal processing circuit 120. Furthermore, the image generation circuit 130 performs rendering processing on the volume image data to generate various 2D ultrasound images for displaying the volume image data on the display 103.

[0020] The image generation circuit 130 stores ultrasound images and ultrasound images that have undergone various image processing in the image memory 140. The image memory 140 and the storage circuit 150 are, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or storage devices such as hard disks and optical disks.

[0021] The processing circuitry 160 controls the overall processing of the ultrasound diagnostic apparatus 1. Specifically, the processing circuitry 160 controls the processing of the transmission / reception circuitry 110, the signal processing circuitry 120, the image generation circuitry 130, and the image memory 140 based on various setting requests input by the operator via the input device 102 and various control programs and various data read from the storage circuitry 150. The processing circuitry 160 controls the display 103 to display an ultrasound image for display generated by the image generation circuitry 130 or an ultrasound image for display stored in the image memory 140.

[0022] The processing circuitry 160 functions as a medical image processing device in this embodiment. FIG. 2 is a block diagram showing the configuration of the processing circuitry 160. As shown in FIG. 2, the processing circuitry 160 executes a 2D ultrasound image acquisition function 161 for registration, a 3D medical image acquisition function 162, a registration function 163, and a display control function 164. The 2D ultrasound image acquisition function 161 has an acquisition function 161A, a derivation function 161B, and a control function 161C. The derivation function 161B is an example of a derivation unit. The control function 161C is an example of a control unit. The registration function 163 is an example of a registration unit.

[0023] Here, the processing functions executed by the 2D ultrasound image acquisition function 161, the 3D medical image acquisition function 162, the registration function 163, and the display control function 164, which are components of the processing circuitry 160 shown in Fig. 2, are recorded in the storage circuitry 150 of the ultrasound diagnostic apparatus 1, for example, in the form of a computer-executable program. The processing circuitry 160 is a processor that realizes the processing function corresponding to each program by reading and executing each program from the storage circuitry 150. In other words, the processing circuitry 160 in a state in which each program has been read has each function shown in the processing circuitry 160 of Fig. 2. The processing contents of the image generation function 141, the detection function 142, the estimation function 143, and the display control function 144 executed by the image processing circuitry 140 will be described later.

[0024] In Figure 2, it is described that each processing function performed by the 2D ultrasound image acquisition function 161, the 3D medical image acquisition function 162, the registration function 163, and the display control function 164 is realized in a single processing circuit 160, but it is also possible to configure a processing circuit by combining multiple independent processors and realize each function by each processor executing a program.

[0025] The term "processor" used in the above description refers to a circuit such as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)). If the processor is a CPU, for example, the processor realizes its function by reading and executing a program stored in the memory circuit 150. On the other hand, if the processor is an ASIC, for example, the program is directly embedded in the processor circuit instead of storing the program in the memory circuit 150. Note that each processor in this embodiment is not limited to being configured as a single circuit, but may be configured as a single processor by combining multiple independent circuits to realize its function. Furthermore, multiple components in FIG. 2 may be integrated into a single processor to realize its function.

[0026] The overall configuration of the ultrasound diagnostic device 1 according to this embodiment has been described above. With this configuration, the ultrasound diagnostic device 1 according to this embodiment performs the following processing so that ultrasound images that facilitate registration can be searched for in a short time. In the ultrasound diagnostic device 1 according to this embodiment, the derivation function 161B derives predefined image evaluation parameters for multiple 2D ultrasound images acquired by the ultrasound probe 101. The control function 161C selects a 2D ultrasound image for registration from the multiple 2D ultrasound images based on the derivation results of the image evaluation parameters. The registration function 163 performs registration between the 3D medical image and the 2D ultrasound image for registration.

[0027] Each processing function of the processing circuit 160 shown in FIG. 2 will be described.

[0028] First, we will explain the 2D ultrasound image acquisition function 161. The 2D ultrasound image acquisition function 161 selects a 2D ultrasound image for registration from the multiple 2D ultrasound images generated by the image generation circuit 130. As described above, the 2D ultrasound image acquisition function 161 has an acquisition function 161A, a derivation function 161B, and a control function 161C.

[0029] The acquisition function 161A acquires a plurality of 2D ultrasound images from the image generation circuit 130 as ultrasound images of the subject P obtained when an ultrasound scan is performed on the subject P. The acquisition function 161A acquires all of the 2D ultrasound images from the image generation circuit 130 without selecting them.

[0030] The derivation function 161B derives image evaluation parameters for all of the 2D ultrasound images acquired by the acquisition function 161A based on predefined image evaluation parameters and a method for deriving the image evaluation parameters. In addition, the derivation function 161B determines the relationship between the derived results of the image evaluation parameters for each 2D ultrasound image and the threshold range corresponding to the image evaluation parameters.

[0031] The image evaluation parameters are indices for evaluating whether a 2D ultrasound image is advantageous for accurate registration. In this embodiment, the image evaluation parameters include, for example, at least one of landmarks, texture metrics, orthogonality metrics, coverage (or overlap), and Doppler blood flow images. These image evaluation parameters are merely examples and are not limited to these. The method for deriving each image evaluation parameter will be described later.

[0032] The control function 161C presents the derivation results of the image evaluation parameters derived by the derivation function 161B to the operator by displaying the derivation results on the display 103. For example, the control function 161C causes a 2D ultrasound image to be displayed together with the derivation results of the image evaluation parameters on the display 103. Specifically, the control function 161C writes the derivation results into the 2D ultrasound image and causes the display 103 to display the image evaluation parameters.

[0033] Furthermore, the control function 161C displays different derivation results on the 2D ultrasound image in different colors depending on the degree of recommendation based on the derivation result and a predetermined threshold range for evaluating the degree of recommendation. Here, the control function 161C may display the derivation results on the 2D ultrasound image in different colors depending on the degree of recommendation, or may highlight a 2D ultrasound image whose derivation result exceeds a predetermined threshold as a suitable 2D ultrasound image on the display 103 depending on the degree of recommendation. This allows the operator to check the 2D ultrasound image for registration by referring to the derivation result displayed on the display 103 and the highlighted 2D ultrasound image.

[0034] Here, the 2D ultrasound image for registration may be selected manually by the operator (for example, by the operator using the input device 102) or automatically by the control function 161C. For example, the control function 161C displays the derivation results on the 2D ultrasound image in different colors depending on the recommendation level, and the operator uses the input device 102 to select the 2D ultrasound image with the highest recommendation level as the 2D ultrasound image for registration. Alternatively, the control function 161C automatically selects a 2D ultrasound image whose derivation result exceeds a predetermined threshold depending on the recommendation level as a suitable 2D ultrasound image and highlights it on the display 103. The control function 161C outputs the selected 2D ultrasound image for registration to the registration function 163.

[0035] The 3D medical image acquisition function 162 acquires a 3D medical image generated by a medical image diagnostic device and outputs it to the registration function 163. Examples of medical image diagnostic devices include an ultrasound diagnostic device, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a SPECT (Single Photon Emission Computed Tomography) device, a PET (Positron Emission Computed Tomography) device, a SPECT-CT device that combines a SPECT device and an X-ray CT device, and a PET-CT device that combines a PET device and an X-ray CT device. The 3D medical image is, for example, any one of a 3D ultrasound image, a 3D computed tomography image, and a 3D magnetic resonance image.

[0036] The registration function 163 performs registration between the 2D ultrasound image for registration output from the control function 161C of the 2D ultrasound image acquisition function 161 and the 3D medical image output from the 3D medical image acquisition function 162. That is, the registration function 163 performs registration as alignment between the 2D ultrasound image for registration and the 3D medical image.

[0037] The display control function 164 causes the display 103 to display the registration result obtained by the registration function 163. That is, the display control function 164 causes the display 103 to display an image obtained by registering the 2D ultrasound image and the 3D medical image for registration.

[0038] As described above, in the ultrasound diagnostic device 1 according to this embodiment, when there are multiple 2D ultrasound images acquired by the 2D ultrasound image acquisition function 161, the derivation function 161B in the 2D ultrasound image acquisition function 161 derives image evaluation parameters for the multiple 2D ultrasound images acquired by the acquisition function 161A based on predefined image evaluation parameters, and the control function 161C selects a suitable 2D ultrasound image from the multiple 2D ultrasound images as a 2D ultrasound image for registration based on the derivation result of the image evaluation parameters, thereby enabling a search for a 2D ultrasound image that facilitates registration in a short time. Therefore, in the ultrasound diagnostic device 1 according to this embodiment, the registration function 163 can be used to quickly and accurately register the 2D ultrasound image for registration with the 3D medical image acquired by the 3D medical image acquisition function 162.

[0039] Here, the processing of the derivation function 161B and the control function 161C of the 2D ultrasound image acquisition function 161 will be described in detail.

[0040] First, the operator pre-sets multiple numerical ranges for evaluating the degree of recommendation for each image evaluation parameter. For example, in the 2D ultrasound image acquisition function 161, the derivation function 161B derives image evaluation parameters for the 2D ultrasound image acquired by the acquisition function 161A to obtain the derivation results of the image evaluation parameters. Then, the derivation results are determined to fall within a numerical range, thereby obtaining the degree of recommendation for the 2D ultrasound image. The derivation results are expressed as normalized numerical values within the interval [0, 1], and the control function 161C displays the 2D ultrasound image including the normalized numerical values on the display 103 as the derivation results. For example, the derivation results can be divided into three normalized numerical ranges: "0 to 0.6," "0.6 to 0.85," and "0.85 to 1." In this case, the highest recommendation level is the interval "0.85 to 1," and the lowest recommendation level is the interval "0 to 0.6."

[0041] For example, the normalized numerical values are displayed as numerical values with colors assigned to each interval, and the control function 161C displays the derivation results in color on the display 103. Specifically, red, yellow, and green are assigned to three intervals: "0 to 0.6," "0.6 to 0.85," and "0.85 to 1," respectively. In this case, the color with the highest recommendation level is green, and the color with the lowest recommendation level is red. For example, if the derivation result as a normalized numerical value is in the interval "0 to 0.6," the control function 161C displays the normalized numerical value in red on the 2D ultrasound image as information indicating that the 2D ultrasound image is not suitable for registration with a 3D medical image. For example, if the derivation result as a normalized numerical value is in the interval "0.6 to 0.85," the control function 161C displays the normalized numerical value in yellow on the 2D ultrasound image as information indicating that the quality of the 2D ultrasound image is average but that it may be used for registration with a 3D medical image. For example, if the derived result as a normalized numerical value is in the range of "0.85 to 1", the control function 161C displays the normalized numerical value in green on the 2D ultrasound image as information indicating that the quality of the 2D ultrasound image is good and that it is suitable for registration with a 3D medical image.

[0042] 3A to 3C are schematic diagrams showing an example of a display of a 2D ultrasound image, in which the control function 161C causes the display 103 to display three 2D ultrasound images together with the derived results as normalized numerical values.

[0043] For example, in FIG. 3A, the information "XXX:0.95" displayed in the upper left corner of the 2D ultrasound image is displayed in green. In FIG. 3B, the information "XXX:0.3" displayed in the upper left corner of the 2D ultrasound image is displayed in red. In FIG. 3C, the information "XXX:0.7" displayed in the upper left corner of the 2D ultrasound image is displayed in yellow. In FIGS. 3A to 3C, the "XXX" in the information is the name of the selected image evaluation parameter, and "0.95," "0.3," and "0.7" are derived results representing normalized values. In FIGS. 3A to 3C, the red, yellow, and green colors used in the information displayed in the upper left corner of the 2D ultrasound image are identified based on the three intervals "0 to 0.6," "0.6 to 0.85," and "0.85 to 1" described above. This allows the operator to easily determine which ultrasound image is suitable based on the colors displayed on the 2D ultrasound image.

[0044] Note that the control function 161C may display only the normalized numerical values "0.95," "0.3," and "0.7" as the derived results on the display 103, and when the operator selects one of the numerical values, the 2D ultrasound image corresponding to the selected numerical value may be used as the 2D ultrasound image for registration. For example, when the operator selects the numerical value "0.95," the control function 161C may use the 2D ultrasound image corresponding to the numerical value "0.95" as the 2D ultrasound image for registration.

[0045] Furthermore, the control function 161C may display information "XXX:0.95", "XXX:0.3", and "XXX:0.7" containing normalized numerical values as derived results in the upper left corner of each of the three 2D ultrasound images, along with information indicating whether or not the information is suitable for registration with a 3D medical image. For example, since the numerical value "0.95" of the information "XXX:0.95" is within the range "0.85 to 1", the control function 161C displays on the display 103 characters indicating that the information "XXX:0.95" is suitable for registration with a 3D medical image, such as information "XXX:0.9 (suitable)".

[0046] In the above examples, several methods for displaying derived results are given, but this embodiment is not limited to these, and other methods may be used as long as they can help the operator distinguish between suitable 2D ultrasound images.

[0047] In the above example, the derived results are normalized to the interval [0, 1] and displayed, but normalization is not necessary.

[0048] FIG. 4 is a schematic diagram showing an example of a 2D ultrasound image, in which a suitable 2D ultrasound image is highlighted on the display 103.

[0049] First, in the upper part of Fig. 4, during real-time scanning of the ultrasound probe, the derivation function 161B derives image evaluation parameters for multiple 2D ultrasound images 200 acquired by the acquisition function 161A, and the control function 161C determines whether the multiple 2D ultrasound images 200 are suitable for registration with a 3D medical image based on the derivation results of the image evaluation parameters. Next, in the lower part of Fig. 4, for example, 2D ultrasound images 200A and 200B among the multiple 2D ultrasound images 200 are considered to be more suitable than the other 2D ultrasound images. In this case, the control function 161C determines that the 2D ultrasound images 200A and 200B are suitable for registration with a 3D medical image, and highlights the 2D ultrasound images 200A and 200B, for example, by highlighting the edges. For example, the edges are frames of the 2D ultrasound images 200A and 200B, and the control function 161C performs highlighting to highlight the frames. For example, the edges are landmarks in the 2D ultrasound images 200A and 200B, and the control function 161C performs highlighting to highlight the landmarks.

[0050] The above-described highlighting may be performed when the 2D ultrasound image and the 3D medical image are registered. For example, if the control function 161C determines that only the 2D ultrasound image 200A among the multiple 2D ultrasound images 200 is suitable for registration with the 3D medical image, the 2D ultrasound image 200A is highlighted even after the registration between the 2D ultrasound image 200A and the 3D medical image is performed. Furthermore, for example, the control function 161C may present the 2D ultrasound images 200A and 200B to the operator by displaying them on the display 103, and then select a 2D ultrasound image selected by the operator as the 2D ultrasound image for registration.

[0051] The difference between the example shown in Figure 4 and the examples shown in Figures 3A to 3C is that in the example shown in Figure 4, the control function 161C highlights a suitable 2D ultrasound image based on the derivation result without displaying a specific derivation result (numerical value) on the display 103.

[0052] 3A to 3C, a 2D ultrasound image is presented to the operator by being displayed on the display 103 together with the derivation result, and in the example shown in Fig. 4, a highlighted 2D ultrasound image is presented to the operator, but the 2D ultrasound image does not have to be displayed to the operator. For example, after the control function 161C determines, based on the derivation result of the image evaluation parameter, that one 2D ultrasound image out of the plurality of 2D ultrasound images is a 2D ultrasound image suitable for registration with a 3D medical image, the control function 161C performs registration with the 3D medical image using the suitable 2D ultrasound image as the 2D ultrasound image for registration without presenting the 2D ultrasound image to the operator.

[0053] 5A to 5C are flowcharts showing the selection of 2D ultrasound images for registration as part of the processing of the medical image processing apparatus according to this embodiment.

[0054] Figure 5A corresponds to the example shown in Figures 3A to 3C, and is a process in which, for example, the processing circuit 160 presents multiple 2D ultrasound images together with the derived results to the operator by displaying them on the display 103, and then registers the 2D ultrasound image selected by the operator with a 3D medical image.

[0055] First, step S101 in Fig. 5A is a step executed by the processing circuitry 160 by calling up programs corresponding to the acquisition function 161A and the 3D medical image acquisition function 162 from the storage circuitry 150. In step S101, the acquisition function 161A of the 2D ultrasound image acquisition function 161 acquires, from the image generation circuitry 130, a plurality of 2D ultrasound images that have been transmitted and processed via the ultrasound probe 101 as ultrasound images of the subject P obtained when an ultrasound scan is performed on the subject P. In addition, the 3D medical image acquisition function 162 acquires a 3D medical image generated by a medical image diagnostic device.

[0056] 5A is a step in which the processing circuitry 160 calls and executes a program corresponding to the derivation function 161B from the storage circuitry 150. In step S102, the derivation function 161B of the 2D ultrasound image acquisition function 161 derives image evaluation parameters for the multiple 2D ultrasound images acquired by the acquisition function 161A based on the image evaluation parameters specified in advance, and obtains numerical values that are the derivation results.

[0057] Next, step S103A in Fig. 5A is a step in which the processing circuitry 160 calls and executes a program corresponding to the control function 161C from the storage circuitry 150. In step S103A, the control function 161C presents a plurality of 2D ultrasound images including numerical values as the derivation results of the image evaluation parameters to the operator by displaying them on the display 103. At this time, the control function 161C sets the 2D ultrasound image selected by the operator as the 2D ultrasound image for registration.

[0058] 5A is a step in which the processing circuitry 160 calls up programs corresponding to the registration function 163 and the display control function 164 from the storage circuitry 150 and executes the programs. In step S103B, the registration function 163 registers the 2D ultrasound image selected by the operator as the 2D ultrasound image for registration with the 3D medical image acquired by the 3D medical image acquisition function 162. The display control function 164 displays on the display 103 an image in which the 2D ultrasound image for registration and the 3D medical image are registered.

[0059] Figure 5B corresponds to the example shown in Figure 4, and is a process in which, for example, the processing circuit 160 presents a highlighted 2D ultrasound image to an operator and then registers the 2D ultrasound image selected by the operator with a 3D medical image.

[0060] First, in FIG. 5B, steps S101 and S102 are executed, similarly to FIG. 5A. Next, step S104A in FIG. 5B is a step executed by the processing circuitry 160 calling up a program corresponding to the control function 161C from the storage circuitry 150. In step S104A, the control function 161C determines whether or not multiple 2D ultrasound images are suitable for registration with a 3D medical image, based on the derived image evaluation parameters, and presents the suitable 2D ultrasound image to the operator by highlighting it on the display 103 compared to the other 2D ultrasound images. At this time, the control function 161C sets the 2D ultrasound image selected by the operator as the 2D ultrasound image for registration.

[0061] Next, step S104B in FIG. 5B is a step in which the processing circuitry 160 calls up programs corresponding to the registration function 163 and the display control function 164 from the storage circuitry 150 and executes the program. In step S104B, the registration function 163 registers the 2D ultrasound image selected by the operator as the 2D ultrasound image for registration with the 3D medical image acquired by the 3D medical image acquisition function 162. The display control function 164 displays an image in which the 2D ultrasound image for registration and the 3D medical image are registered on the display 103. Here, even after the registration of the 2D ultrasound image and the 3D medical image is performed, for example, the control function 161C may perform an emphasis display, such as highlighting the edges of the 2D ultrasound image.

[0062] FIG. 5C illustrates a process in which, for example, the processing circuit 160 performs registration with a 3D medical image using a suitable 2D ultrasound image as a 2D ultrasound image for registration without presenting the suitable 2D ultrasound image to the operator.

[0063] First, in Fig. 5C, steps S101 and S102 are executed similarly to Fig. 5A. Next, step S105A in Fig. 5C is a step executed by the processing circuitry 160 calling a program corresponding to the control function 161C from the storage circuitry 150. In step S105A, when the control function 161C determines, based on the derivation result of the image evaluation parameter, that one 2D ultrasound image among the plurality of 2D ultrasound images is a 2D ultrasound image suitable for registration with a 3D medical image, it selects the suitable 2D ultrasound image as the 2D ultrasound image for registration without presenting it to the operator.

[0064] 5C is a step in which the processing circuitry 160 calls up and executes programs corresponding to the registration function 163 and the display control function 164 from the storage circuitry 150. In step S105B, the registration function 163 registers the 2D ultrasound image selected by the control function 161C with the 3D medical image acquired by the 3D medical image acquisition function 162. The display control function 164 causes the display 103 to display an image in which the 2D ultrasound image for registration and the 3D medical image are registered.

[0065] According to this embodiment, in the 2D ultrasound image acquisition function 161, the derivation function 161B derives image evaluation parameters for multiple 2D ultrasound images acquired by the acquisition function 161A based on predefined image evaluation parameters, and the control function 161C selects a suitable 2D ultrasound image from the multiple 2D ultrasound images as a 2D ultrasound image for registration based on the derivation results of the image evaluation parameters, thereby making it possible to search for a 2D ultrasound image that is advantageous for improving registration accuracy in a short period of time.

[0066] Furthermore, in this embodiment, the number of 2D ultrasound images selected as the 2D ultrasound image for registration may not be one, but multiple. That is, when multiple 2D ultrasound images are selected as the 2D ultrasound images for registration by the control function 161C, the registration function 163 registers the multiple 2D ultrasound images selected by the control function 161C with the 3D medical image acquired by the 3D medical image acquisition function 162. In this case, the accuracy of registration can be further improved.

[0067] In addition, in this embodiment, the derivation function 161B derives image evaluation parameters for multiple 2D ultrasound images, and then displays the 2D ultrasound images on the display 103 together with the derivation results of the image evaluation parameters, or presents a suitable (high registration accuracy) 2D ultrasound image to the operator so that it is highlighted on the display 103 and the operator can confirm it, thereby making it easier for the operator to select a 2D ultrasound image suitable for registration.

[0068] In addition, in this embodiment, the derivation function 161B may automatically select a suitable 2D ultrasound image as a 2D ultrasound image for registration based on the derivation results of the image evaluation parameters without presenting the suitable 2D ultrasound image to the operator.In this case, since the confirmation process by the operator is not required, registration can be completed in a shorter time.

[0069] The image evaluation parameters used in this embodiment will be described in detail below with examples, where the image evaluation parameters used in this embodiment include at least one of a landmark, a texture metric, an orthogonality metric, a coverage, and a Doppler blood flow image.

[0070] (Landmark) Landmarks are anatomically distinctive features that are key points in surgery. For example, if the organ to be treated is the liver, landmarks include the convergence points of the portal vein and hepatic vein, and the origins of the portal vein and hepatic vein. Figures 6A and 6B show 2D ultrasound images containing landmarks. In the examples shown in Figures 6A and 6B, the convergence points of the portal vein and hepatic vein in the liver are shown as dots, respectively. Landmarks can be detected in medical images of different modes using detection algorithms such as deep neural networks.

[0071] Therefore, the control function 161C selects a 2D ultrasound image including landmarks as a 2D ultrasound image for registration. The registration function 163 detects landmarks from the 2D ultrasound image and 3D medical image for registration, and registers the 2D ultrasound image and the 3D medical image using the detected landmarks. This improves the accuracy of registration. In this manner, in this embodiment, by using landmarks as image evaluation parameters, a 2D ultrasound image suitable for registration with a 3D medical image can be obtained.

[0072] Here, when extracting anatomically significant features as feature amounts, for example, gray scale, feature descriptors, neural networks, etc. are used.

[0073] When grayscale is used, a histogram is created by detecting extreme values from the difference image of smoothed images obtained by filters of different scales, and features are extracted from the changes in grayscale in local or global regions of the image represented by the histogram.

[0074] When a feature descriptor is used, it calculates the difference between images at different scales to extract features that are robust against image scaling, translation, and rotation. Feature descriptors can be expressed as vectors. Common feature descriptors include Harris corner detection, SIFT (Scale-Invariant Feature Transform), and SURF (Speeded-Up Robust Features).

[0075] When a neural network is used, an image is input into the neural network, and feature values are extracted by directly determining the coordinates of the feature points. Examples of neural networks include convolutional neural networks (CNNs) such as the Hourglass network and U-shaped neural networks UNet and nnUNet, as well as regenerative neural networks such as LSTM (Long Short-Term Memory), RNN (Recurrent Neural Network), which is a simplified model of LSTM, and GRU (Gated Recurrent Unit), but are not limited to these examples.

[0076] (Texture Metrics) For example, if a 2D ultrasound image contains a region that can be distinguished by an anatomical name such as "portal vein" or "origin of portal vein," landmarks are used as image evaluation parameters as described above. For regions that cannot be distinguished by an anatomical name in a 2D ultrasound image, texture metrics are used as image evaluation parameters, for example.

[0077] 7A and 7B are diagrams illustrating 2D ultrasound images with different texture structures. The texture metric indicates the number of texture structures in the ultrasound image. The more obvious the texture structures, the more accurate the registration can be. For example, the 2D ultrasound image shown in FIG. 7B has more texture structures than the 2D ultrasound image shown in FIG. 7A. In this case, it is preferable to select the 2D ultrasound image shown in FIG. 7B as the 2D ultrasound image for registration.

[0078] The number of texture structures can be determined using the standard deviation. For example, a central region of the ultrasound image is selected, and the standard deviation SD of the region is calculated using Equation 1.

[0079]

number

[0080] In the formula, I j is the pixel intensity, I is the average pixel intensity, and N is the number of pixels. After calculating the standard deviation for each region, each standard deviation is normalized to the interval [0,1].

[0081] This allows the operator to select a scan plane (i.e., a 2D ultrasound image) in which the normalized value of the texture metric is closer to 1. For example, the control function 161C presents a plurality of 2D ultrasound images including the normalized values of the texture metric to the operator by displaying them on the display 103. At this time, the control function 161C sets the 2D ultrasound image selected by the operator as the 2D ultrasound image for registration. In this way, in this embodiment, by using the texture metric as the image evaluation parameter, a 2D ultrasound image including more texture structure can be obtained as the 2D ultrasound image for registration.

[0082] In the above example, the standard deviation of the brightness of the 2D ultrasound image is used as a texture metric, which is an example of an image evaluation parameter, but this is not limiting. The texture metric, which is an example of an image evaluation parameter, may be set based on at least one of the brightness gradient and brightness difference in the 2D ultrasound image. Furthermore, the standard deviation, brightness gradient, and brightness difference used as the texture metric may be calculated for the entire image in each ultrasound image, or may be calculated for a partial region (such as a central region). The parameters of the standard deviation, brightness gradient, and brightness difference as texture metrics ultimately indicate the boundaries and contours of organs, and the process of calculating these parameters corresponds to the process of generating an edge-enhanced image from an original image.

[0083] (orthogonal metric) When performing registration using multiple 2D ultrasound images, it is desirable to select multiple 2D ultrasound images with large differences in scan direction, i.e., scan directions that are closer to orthogonal. This is because, when the scan direction is maintained, differences in the internal texture structure may not be apparent when scanning by translating the probe, whereas scanning in different directions allows for the observation of richer texture structures.

[0084] An orthogonality metric is used to determine the orthogonality between the scan planes. The orthogonality metric Orth can be derived using the following Equation 2.

[0085]

number

[0086] where i represents the i-th selected 2D ultrasound image, and N i represents the orthogonal component of the i-th selected ultrasound image, N is the number of 2D ultrasound images, and N' is the orthogonal component of the current 2D ultrasound image. It is desirable to select ultrasound images with a large orthogonality metric Orth whenever possible.

[0087] The following description will be made with reference to FIG. 8. As shown in FIG. 8, each scan plane is transformed into a magnetic coordinate system. For example, the magnetic coordinates are obtained from a magnetic sensor attached to the ultrasound probe 101. In FIG. 8, a 2D ultrasound image is denoted as "2D US." I1, I2, I', and I" represent multiple scan planes (i.e., 2D ultrasound images) having different scan directions. I1 and I2 are the selected 2D ultrasound images, N1 and N2 are orthogonal components of the selected I1 and I2, I' and I" are 2D ultrasound images that are candidates for selection, and N' and N" are orthogonal components of the selected I' and I" candidates. As a result of comparing I' and I", it is found that I' is more orthogonal to the selected I1 and I2. Therefore, it is preferable to select I' as the 2D ultrasound image for registration without selecting I"

[0088] After the orthogonal metrics Orth are calculated, each orthogonal metric Orth is normalized to the interval [0, 1].

[0089] This allows the operator to select a scan plane in which the normalized value of the orthogonality metric is closer to 1. For example, the control function 161C presents a plurality of 2D ultrasound images including the normalized values of the orthogonality metric to the operator by displaying them on the display 103. At this time, the control function 161C sets the 2D ultrasound image selected by the operator as the 2D ultrasound image for registration. In this way, in this embodiment, by using the orthogonality metric as the image evaluation parameter, it is possible to obtain, as the 2D ultrasound image for registration, a 2D ultrasound image in which the scanning direction of each 2D ultrasound image is closer to orthogonal.

[0090] (coverage) The 2D ultrasound images for registration must cover as large a region of interest (ROI) as possible. The region of interest is the area that the operator particularly wants to observe; for example, in liver ablation surgery, the region of interest refers to the entire liver. The coverage represents the size of the common area between the closed region formed in three-dimensional space by multiple 2D ultrasound images and the region of interest. It is preferable to use multiple 2D ultrasound medical images in which the closed region formed in three-dimensional space occupies a large proportion of the entire region of interest as ultrasound images for registration.

[0091] The method for determining the coverage will be described below with reference to Figures 9A and 9B. In Figures 9A and 9B, each scan plane is converted into a magnetic coordinate system. For example, the magnetic coordinates are obtained from a magnetic sensor attached to the ultrasound probe 101. In Figures 9A and 9B, the 2D ultrasound image is represented as "2D US" and the region of interest is represented as "ROI." Here, the closed region refers to the region identified by connecting the intersections of the outer scan plane and the region of interest in the magnetic coordinate system. I1, I2, and I3 shown in Figure 9A and I1, I2, and I4 shown in Figure 9B represent multiple scan planes with different scanning directions. I1 and I2 shown in Figures 9A and 9B are selected 2D ultrasound images, and I3 and I4 are candidate ultrasound images. The mesh patterns in Figures 9A and 9B represent the intersections between the closed regions formed in three-dimensional space by each 2D ultrasound image and the region of interest (ROI). Here, the size of the common area formed by I1, I2, and I4 and the region of interest ROI shown in Figure 9B is larger than the size of the common area formed by I1, I2, and I3 and the region of interest ROI shown in Figure 9A. Therefore, it is preferable to select I4 without selecting I3.

[0092] The coverage measure may similarly be achieved by a detection algorithm, and after obtaining the coverage of each scan plane for the region of interest, the coverage can be normalized to the interval [0,1].

[0093] This allows the operator to select a scan plane with a coverage normalization value closer to 1. For example, the control function 161C presents a plurality of 2D ultrasound images including the coverage normalization values to the operator by displaying them on the display 103. At this time, the control function 161C sets the 2D ultrasound image selected by the operator as the 2D ultrasound image for registration. In this way, in this embodiment, by using coverage as an image evaluation parameter, a plurality of 2D ultrasound images covering a larger region of interest can be obtained.

[0094] (Doppler blood flow image) Generally, the major blood vessels in the human body all have clear structural features, and the clearer the structural features, the better the registration effect. By using Doppler blood flow images as 2D ultrasound images, it is easy to observe whether the 2D ultrasound images contain blood flow information.

[0095] This allows the operator to select a 2D ultrasound image including blood flow information. For example, the control function 161C presents a plurality of 2D ultrasound images including blood flow information to the operator by displaying them on the display 103. At this time, the control function 161C sets the 2D ultrasound image selected by the operator as the 2D ultrasound image for registration. In this way, in this embodiment, by using a Doppler blood flow image as an image evaluation parameter, registration can be easily performed and the accuracy of registration can be improved.

[0096] In this embodiment, landmarks, texture metrics, orthogonality metrics, coverage, Doppler blood flow images, and the like are listed as image evaluation parameters, but the image evaluation parameters are not limited to these.

[0097] When a 2D ultrasound image for registration is selected using only one of the image evaluation parameters, the control function 161C displays information indicating the image evaluation parameter used on the 2D ultrasound image. For example, when only a landmark is used as the image evaluation parameter, the control function 161C displays "landmark" on the 2D ultrasound image as information indicating the image evaluation parameter used. Furthermore, when a 2D ultrasound image for registration is selected using multiple image evaluation parameters, the control function 161C displays information indicating the multiple image evaluation parameters used on the 2D ultrasound image. For example, weighting coefficients are assigned to the multiple image evaluation parameters, and when a landmark, texture metric, and orthogonality metric are used as the image evaluation parameters, the control function 161C displays "landmark: 1; texture metric: 0.7; orthogonality metric: 0.5" on the 2D ultrasound image as information indicating the multiple image evaluation parameters used.

[0098] Here, "1", "0.7", and "0.5" are weighting coefficients assigned to the "landmark", "texture metric", and "orthogonality metric", respectively. Specifically, when using multiple image evaluation parameters, for example, the operator assigns different weighting coefficients (weights) to the multiple image evaluation parameters based on the type of surgery, the area of interest, etc., and the control function 161C calculates weighting scores based on the weighting coefficients assigned to the multiple image evaluation parameters, and selects a 2D ultrasound image for registration based on the weighting scores.

[0099] The weighted score S for each 2D ultrasound image k The calculation formula is shown in Equation 3.

[0100]

number

[0101] In the formula, M i is the evaluation parameter of the i-th 2D ultrasound image, and W iis the weight of the i-th 2D ultrasound image, and N is the number of image evaluation parameters. The control function 161C calculates the total weighting score using a weighting calculation method, and selects a 2D ultrasound image for registration based on the calculated weighting score. In this case, the control function 161C may display information indicating the final weighting score on the 2D ultrasound image as information indicating the multiple image evaluation parameters used. For example, the control function 161C displays only "weighting score: 0.75" on the 2D ultrasound image as information indicating the final weighting score. Here, the closer the weighting score is to 1, the better.

[0102] As a result, the control function 161C calculates weighting scores based on the weighting factors assigned to the multiple image evaluation parameters, thereby more comprehensively evaluating each 2D ultrasound image, and selects a 2D ultrasound image for registration based on the evaluation results. In this way, in this embodiment, by assigning weighting factors to the multiple image evaluation parameters, it is possible to select a 2D ultrasound image that is more suitable for registration.

[0103] (Example 1) Here, as a specific example of registration, registration using landmarks will be described.

[0104] In this case, the registration function 163 detects landmarks from multiple 2D ultrasound images and 3D medical images for registration. By using landmarks, registration between multiple 2D ultrasound images and 3D medical images can be easily achieved.

[0105] FIG. 10 is a diagram for explaining registration using landmarks. Note that I1, I2, and I3 represent 2D ultrasound images, P1, P2, and P3 are landmarks in each 2D ultrasound image, and Q1, Q2, and Q3 are landmarks in the 3D medical image. As shown in the upper part of FIG. 10, the registration function 163 uses a detection algorithm to detect landmarks P1, P2, and P3 in the 2D ultrasound image and landmarks Q1, Q2, and Q3 in the 3D medical image. Then, the registration function 163 matches the landmarks P1, P2, and P3 with the landmarks Q1, Q2, and Q3 to generate dot pairs of corresponding landmarks. <P i , Q i Then, the registration function 163 obtains <P i , Q i Using the dot pairs, transformation parameters are generated that indicate the correspondence between the landmarks Q1, Q2, and Q3 in the 3D medical image and the corresponding landmarks P1, P2, and P3 in each 2D ultrasound image, and the registration function 163 registers each 2D ultrasound image with the 3D medical image based on the transformation parameters. Note that i represents the number of dot pairs and is an integer equal to or greater than 1, and in the above example, i=3.

[0106] 11 is a flowchart showing how to perform registration using landmarks. The registration of a 2D ultrasound image and a 3D medical image in Example 1 will be described below with reference to FIG.

[0107] 11, first, in step S201, the same processing as in step S101 described above is performed. That is, the acquisition function 161A of the 2D ultrasound image acquisition function 161 acquires from the image generation circuit 130 a plurality of 2D ultrasound images that have been transmitted and processed via the ultrasound probe 101 as ultrasound images of the subject P obtained when an ultrasound scan is performed on the subject P.

[0108] Next, in step S202, processing corresponding to the aforementioned steps S102 and S103A or the aforementioned steps S102 and S104A is performed. That is, the derivation function 161B of the 2D ultrasound image acquisition function 161 derives image evaluation parameters for the multiple 2D ultrasound images acquired by the acquisition function 161A based on the image evaluation parameters specified in advance, and obtains numerical values that are the derivation results of the image evaluation parameters. Then, the control function 161C presents information for specifying 2D ultrasound images for registration to the operator by displaying it on the display 103. The information includes at least one of a 2D ultrasound image together with the derivation results of the image evaluation parameters and a highlighted 2D ultrasound image. For example, the control function 161C presents the multiple 2D ultrasound images including numerical values as the derivation results of the image evaluation parameters to the operator by displaying them on the display 103. For example, the control function 161C determines whether or not multiple 2D ultrasound images are suitable for registration with a 3D medical image based on the derived image evaluation parameters, and presents the suitable 2D ultrasound image to the operator by highlighting it compared to the other 2D ultrasound images on the display 103. At this time, the control function 161C sets the 2D ultrasound image selected by the operator as the 2D ultrasound image for registration and outputs it to the registration function 163.

[0109] Note that processing equivalent to steps S102 and S105A described above may be performed. That is, the derivation function 161B of the 2D ultrasound image acquisition function 161 derives image evaluation parameters for the multiple 2D ultrasound images acquired by the acquisition function 161A based on the image evaluation parameters specified in advance, and obtains numerical values that are the derivation results of the image evaluation parameters. Then, based on the derivation results of the image evaluation parameters, the control function 161C selects, from the multiple 2D ultrasound images, a 2D ultrasound image that is suitable for registration with the 3D medical image as the 2D ultrasound image for registration, and outputs it to the registration function 163.

[0110] Thereafter, in step S203, the registration function 163 detects landmarks for registration for each of the 2D ultrasound images for registration output from the control function 161C.

[0111] Meanwhile, in step S204, the 3D medical image acquisition function 162 acquires a 3D medical image captured by the medical image diagnostic device and outputs it to the registration function 163. Thereafter, in step S205, the registration function 163 detects landmarks for registration in the 3D medical image output from the control function 161C. Here, the detection of landmarks in the 2D ultrasound image and the detection of landmarks in the 3D medical image may be performed in parallel or in chronological order, and the timing of the landmark detection is not limited. When the detection of landmarks in the 2D ultrasound image and the detection of landmarks in the 3D medical image are performed in chronological order, for example, the display control function 164 displays the 3D medical image on the display 103 by highlighting or otherwise emphasizing the landmarks in the 3D medical image, and the operator operates the ultrasound probe 101 while referring to the landmarks in the 3D medical image displayed on the display 103, thereby executing the above-mentioned steps S201 to S203.

[0112] Next, in step S206, the registration function 163 matches the landmarks P1, P2, and P3 in the 2D ultrasound image with the landmarks Q1, Q2, and Q3 in the 3D medical image to generate dot pairs of corresponding landmarks. <P i , Q i Then, the registration function 163 obtains <P i , Q i Using the dot pairs, transformation parameters are generated that indicate the correspondence between the landmarks Q1, Q2, and Q3 in the 3D medical image and the corresponding landmarks P1, P2, and P3 in each 2D ultrasound image. Then, in step S207, the registration function 163 registers the 2D ultrasound image and the 3D medical image using the generated transformation parameters.

[0113] According to specific example 1, registration between 2D ultrasound images and 3D medical images can be easily achieved, thereby reducing the time required for registration and improving the accuracy of registration, thereby reducing the burden on patients during surgery.

[0114] (Example 2) Here, when a texture metric is used as an image evaluation parameter, a specific example will be described in which the texture metric is set based on the brightness gradient among the brightness gradients and brightness differences in multiple 2D ultrasound images. Figures 12A and 12B show specific examples in which multiple 2D ultrasound images are selected as 2D ultrasound images for registration, and "2DUS image" in the figures indicates the 2D ultrasound image.

[0115] For example, when an operator performs an ultrasound scan on a subject P, multiple 2D ultrasound images are acquired by the acquisition function 161A. At this time, the derivation function 161B derives image evaluation parameters for the multiple 2D ultrasound images acquired by the acquisition function 161A. Specifically, if the brightness gradient of a 2D ultrasound image is large, the derivation function 161B sets a large value to the derived result of the image evaluation parameter for a 2D ultrasound image with a large brightness gradient among the multiple 2D ultrasound images, and sets a small value to the derived result of the image evaluation parameter for a 2D ultrasound image with a small brightness gradient.

[0116] As described above, the derivation results are expressed as normalized numerical values and normalized within the interval [0, 1]. Therefore, as shown in Fig. 12A, for example, the derivation function 161B sets the 2D ultrasound image 200 with the largest image brightness gradient as the reference 2D ultrasound image, sets a numerical value of "0.9" to the derivation results of the image evaluation parameters of the reference 2D ultrasound image, and sets numerical values smaller than the numerical value "0.9" to the derivation results of the image evaluation parameters of 2D ultrasound images 200 other than the reference 2D ultrasound image. Specifically, the more distant the derivation results of the image evaluation parameters of 2D ultrasound images 200 other than the reference 2D ultrasound image, the smaller the numerical value set. 12A, the derived results of the image evaluation parameters for the 2D ultrasound images 200 that are one image away from the reference 2D ultrasound image in the front and back are set to the numerical values "0.7" and "0.8", respectively, and the derived results of the image evaluation parameters for the 2D ultrasound images 200 that are three image away from the reference 2D ultrasound image in the front and back are set to the numerical values "0.3" and "0.1", respectively. Here, when the control function 161C causes the derivation results to be displayed in color on the display 103, the control function 161C causes the derivation results to be displayed on the display 103 in red, yellow, and green, which are respectively assigned to the three intervals "0 to 0.6", "0.6 to 0.85", and "0.85 to 1", as described above.

[0117] Since the multiple 2D ultrasound images obtained by ultrasound scanning are image data arranged along a certain axis, the control function 161C selects, as the multiple 2D ultrasound images, a reference 2D ultrasound image 200 having the largest image brightness gradient and a predetermined number of 2D ultrasound images arranged before and after the reference 2D ultrasound image. For example, the multiple 2D ultrasound images are an odd number of 2D ultrasound images equal to or greater than three. In the example shown in FIG. 12B, when the predetermined number is three, the control function 161C selects seven 2D ultrasound images as the multiple 2D ultrasound images 201-207. Of the first to seventh multiple 2D ultrasound images 201-207, the fourth 2D ultrasound image 204 is a reference 2D ultrasound image serving as a reference and is a 2D ultrasound image having a large image brightness gradient.

[0118] The control function 161C assigns weighting coefficients (weights) to the derivation results of the image evaluation parameters of the multiple 2D ultrasound images 201-207, and derives overall image evaluation parameters for the multiple 2D ultrasound images 201-207 based on the derivation results of the image evaluation parameters and the weighting coefficients. As shown in Fig. 12B, for example, the control function 161C assigns a weighting coefficient of "0.5" to the derivation result of the image evaluation parameters of the 2D ultrasound image 204, which is the reference 2D ultrasound image, of "0.9," and assigns weighting coefficients smaller than the weighting coefficient "0.5" to the derivation results of the image evaluation parameters of the 2D ultrasound images 201-203 and 205-207 other than the 2D ultrasound image 204. Specifically, the further away from the 2D ultrasound image 204 is the deriving result of the image evaluation parameters of the 2D ultrasound images 201-203 and 205-207, the smaller the weighting coefficients assigned to them. In the example shown in Figure 12B, a weighting factor of "0.25" is assigned to the derived results of the image evaluation parameters of 2D ultrasound images 203 and 205, and a weighting factor of "0.02" is assigned to the derived results of the image evaluation parameters of 2D ultrasound images 201 and 207.

[0119] Overall image evaluation parameter Q vol The calculation formula is shown in Equation 4.

[0120]

number

[0121] In the formula, Q i is the evaluation parameter of the i-th 2D ultrasound image, and W i is the weight of the i-th 2D ultrasound image, and N is the number of 2D ultrasound images. Based on the result of deriving the overall image evaluation parameter, the control function 161C selects the 2D ultrasound image 204, which is the reference 2D ultrasound image, or a plurality of 2D ultrasound images 201 to 207 as 2D ultrasound images for registration. In the example shown in FIG. 12B, the control function 161C derives the overall image evaluation parameter Q vol For example, when the calculated overall image evaluation parameter Q volThe control function 161C sets the overall image evaluation parameter Q vol is equal to or greater than the preset image evaluation parameter, the plurality of 2D ultrasound images 201 to 207 are selected as 2D ultrasound images for registration. For example, if the preset image evaluation parameter for the liver is "0.8", the overall image evaluation parameter Q vol is equal to or greater than the set image evaluation parameter, the control function 161C selects the multiple 2D ultrasound images 201 to 207 as 2D ultrasound images for registration. vol If ≡(x,y) is less than the set image evaluation parameter, control function 161C selects only 2D ultrasound image 204, which is the reference 2D ultrasound image, as the 2D ultrasound image for registration.

[0122] Then, when multiple 2D ultrasound images 201 to 207 are selected as 2D ultrasound images for registration, the registration function 163 reconstructs a 3D medical image (volume data) based on the multiple 2D ultrasound images 201 to 207, and performs registration using the reconstructed 3D medical image and the 3D medical image acquired by the 3D medical image acquisition function 162.

[0123] In this way, in Specific Example 2 as well, registration between a 2D ultrasound image and a 3D medical image can be easily achieved, the time required for registration can be shortened, and the accuracy of registration can be improved, thereby reducing the burden on the patient during surgery. Here, Specific Example 2 gives an example in which texture metrics are used, but the registration function 163 may detect landmarks from the 2D ultrasound image for registration and the 3D medical image acquired by the 3D medical image acquisition function 162, and register the 2D ultrasound image for registration with the 3D medical image using the detected landmarks.

[0124] (Other embodiments) This embodiment is not limited to the above-described embodiment. For example, the processing circuitry 160 may be a workstation installed separately from the ultrasound diagnostic apparatus 1. In this case, the workstation has a processing circuit similar to the processing circuitry 160 as a medical image processing device and executes the above-described processing.

[0125] Furthermore, the components of each device illustrated in the embodiments are conceptual functional units and do not necessarily have to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to that illustrated, and all or part of each device can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0126] Furthermore, the processing method (medical image processing method) described in the above embodiment can be realized by executing a prepared program on a computer such as a personal computer or a workstation. This program can be distributed via a network such as the Internet. This program can also be recorded on a non-transitory computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO, or DVD, and executed by being read from the recording medium by a computer.

[0127] According to at least one of the embodiments described above, ultrasound images that facilitate registration can be searched for in a short time.

[0128] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0129] 1. Ultrasound diagnostic equipment 161B Derivation function 161C Control Function 163 Registration Function

Claims

1. a derivation unit that derives predefined image evaluation parameters for a plurality of two-dimensional ultrasound images acquired by the ultrasound probe; a control unit that selects a two-dimensional ultrasound image for registration from the plurality of two-dimensional ultrasound images based on the derived image evaluation parameters; a registration unit that performs registration between the three-dimensional medical image and the two-dimensional ultrasound image for registration; An ultrasound diagnostic device comprising:

2. a landmark detection unit that detects landmarks from the two-dimensional ultrasound image for registration and the three-dimensional medical image; Further provided with the registration unit performs the registration based on landmarks in the three-dimensional medical image and corresponding landmarks in the two-dimensional ultrasound image for registration; The ultrasonic diagnostic apparatus according to claim 1 .

3. The control unit causes a display unit to display a two-dimensional ultrasound image together with the derived result.

3. The ultrasonic diagnostic apparatus according to claim 1.

4. the control unit displays the derived result differently depending on the recommendation level. The ultrasonic diagnostic apparatus according to claim 3 .

5. The control unit highlights the two-dimensional ultrasound image in which the derived result exceeds a predetermined threshold.

3. The ultrasonic diagnostic apparatus according to claim 1.

6. The control unit automatically selects a two-dimensional ultrasound image for which the derivation result exceeds a predetermined threshold as the two-dimensional ultrasound image for registration.

3. The ultrasonic diagnostic apparatus according to claim 1.

7. At least one of landmarks, texture metrics, orthogonality metrics, coverage, and Doppler blood flow images is used as the image evaluation parameter. The ultrasonic diagnostic apparatus according to claim 1 .

8. When the texture metric is used as the image evaluation parameter, the control unit selects a two-dimensional ultrasound image including a larger amount of texture structure as the two-dimensional ultrasound image for registration. The ultrasonic diagnostic apparatus according to claim 7.

9. When the orthogonality metric is used as the image evaluation parameter, the control unit selects, as the two-dimensional ultrasound images for the registration, two-dimensional ultrasound images whose scanning directions are nearly orthogonal to each other. The ultrasonic diagnostic apparatus according to claim 7.

10. When the degree of coverage is used as the image evaluation parameter, the control unit selects, from the plurality of two-dimensional ultrasound images, a two-dimensional ultrasound image in which a closed region formed in the three-dimensional space occupies a large proportion of the entire region of interest as the two-dimensional ultrasound image for registration. The ultrasonic diagnostic apparatus according to claim 7.

11. The control unit displays a normalized numerical value as the derived result. The ultrasonic diagnostic apparatus according to claim 1 .

12. The three-dimensional medical image is any one of a three-dimensional ultrasound image, a three-dimensional computed tomography image, and a three-dimensional magnetic resonance image. The ultrasonic diagnostic apparatus according to claim 1 .

13. The landmark is a site having an anatomically prominent feature. The ultrasonic diagnostic apparatus according to claim 2 .

14. the landmark detection unit generates transformation parameters indicating a correspondence between landmarks in the three-dimensional medical image and corresponding landmarks in the two-dimensional ultrasound image; the registration unit performs the registration based on the transformation parameters. The ultrasonic diagnostic apparatus according to claim 2 .

15. the registration unit performs the registration using the plurality of two-dimensional ultrasound images and the three-dimensional medical image when the control unit selects the plurality of two-dimensional ultrasound images as two-dimensional ultrasound images for registration; The ultrasonic diagnostic apparatus according to claim 1 .

16. When the control unit selects the plurality of two-dimensional ultrasound images as the two-dimensional ultrasound images for the registration, the registration unit reconstructs a three-dimensional medical image based on the plurality of two-dimensional ultrasound images, and performs the registration using the reconstructed three-dimensional medical image and the three-dimensional medical image. The ultrasonic diagnostic apparatus according to claim 1 .

17. The control unit a reference two-dimensional ultrasound image which is the two-dimensional ultrasound image having the largest image evaluation parameter, and a predetermined number of two-dimensional ultrasound images arranged before and after the reference two-dimensional ultrasound image, are selected as the plurality of two-dimensional ultrasound images; weights are assigned to the derived results of the image evaluation parameters of the plurality of two-dimensional ultrasound images so that the weights decrease as the image is further away from the reference two-dimensional ultrasound image, and overall image evaluation parameters of the plurality of two-dimensional ultrasound images are derived based on the derived results and the weights; If the overall image evaluation parameter is equal to or greater than a set image evaluation parameter, selecting the plurality of two-dimensional ultrasound images as two-dimensional ultrasound images for the registration. The ultrasonic diagnostic apparatus according to claim 16.

18. When a texture metric is used as the image evaluation parameter, the image evaluation parameter is set based on at least one of a brightness gradient and a brightness difference in the plurality of two-dimensional ultrasound images. The ultrasonic diagnostic apparatus according to claim 17.

19. a derivation unit that derives predefined image evaluation parameters for a plurality of two-dimensional ultrasound images acquired by the ultrasound probe; a control unit that selects a two-dimensional ultrasound image for registration from the plurality of two-dimensional ultrasound images based on the derived image evaluation parameters; a registration unit that performs registration between the three-dimensional medical image and the two-dimensional ultrasound image for registration; A medical image processing device comprising:

20. deriving predefined image evaluation parameters for a plurality of two-dimensional ultrasound images acquired by the ultrasound probe; selecting a two-dimensional ultrasound image for registration from the plurality of two-dimensional ultrasound images based on the derived image evaluation parameters; performing registration between the three-dimensional medical image and the two-dimensional ultrasound image for registration; A medical image processing method comprising:

21. deriving predefined image evaluation parameters for a plurality of two-dimensional ultrasound images acquired by the ultrasound probe; selecting a two-dimensional ultrasound image for registration from the plurality of two-dimensional ultrasound images based on the derived image evaluation parameters; performing registration between the three-dimensional medical image and the two-dimensional ultrasound image for registration; A program that causes a computer to perform a process.

Images