Image processing device, method for processing image and program

Abstract

To provide a technology that improves estimation accuracy for a reference cross section targeting a three-dimensional image.SOLUTION: An image processing device includes: an image acquisition unit for acquiring a three-dimensional image of a subject; an initial parameter estimation unit for reducing resolution of the three-dimensional image and estimating an initial parameter that is an initial value of a parameter for acquiring a reference cross section; a cross-sectional image acquisition unit for acquiring at least one cross-sectional image from the three-dimensional image on the basis of the initial parameter; and an index estimation unit for estimating an index contributing to correction of the initial parameter for each of the cross-sectional images.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology disclosed herein relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In diagnosis using medical images, a 3D image (volume data) is cut based on a predetermined standard, and the cut surface, a 2D cross section (reference cross section), is displayed on a display etc. to perform a diagnosis. However, manual setting of the reference cross section requires the doctor to find anatomical landmarks (feature points) that serve as clues in the 3D space, which places a heavy burden on the doctor.

[0003] In order to solve this problem, various techniques for automatically estimating a reference cross section have been proposed. Patent Document 1 discloses a method for estimating a reference cross section from a three-dimensional moving image consisting of multiple time phases in which a cardiac region is imaged. In Patent Document 1, a reference cross section is estimated at a specific reference time phase among the multiple time phases, and the reference cross section is moved and rotated by inter-time phase tracking to obtain a reference cross section of a time phase other than the reference time phase. In Patent Document 2, candidates for the reference cross section (candidate cross sections) are obtained from an image in which the cardiac region is imaged, and feature point coordinates are calculated from multiple cross sections obtained by translating and rotating the candidate cross sections. Then, the candidate cross sections are corrected by combining these feature points. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-161213 A [Patent Document 2] International Publication No. 2016 / 195110 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the case of estimating the reference cross section using the methods described in the prior art, there are cases where an inference result with sufficient accuracy cannot be obtained.

[0006] The technology of the present disclosure has been made in view of the above, and has an object to provide a technology for improving the estimation accuracy of reference cross sections targeted at three-dimensional images. [Means for solving the problem]

[0007] In order to achieve the above object, the image processing device according to the present disclosure comprises: an image acquisition unit for acquiring a three-dimensional image of a subject; an initial parameter estimation unit that estimates initial parameters, which are initial values ​​of parameters for obtaining a reference cross section, by reducing the resolution of the three-dimensional image; a cross-sectional image acquisition unit that acquires at least one cross-sectional image from the three-dimensional image based on the initial parameters; an index estimation unit that estimates an index that contributes to correction of the initial parameters for each of the cross-sectional images; The image processing device includes:

[0008] Furthermore, the technology disclosed herein can be understood as an image processing method including at least a part of the above-mentioned processing, a program for causing a computer to execute the method, or a computer-readable recording medium on which such a program is non-temporarily recorded. The above-mentioned configurations and processing can be implemented in combination with each other as long as no technical contradiction occurs. Cut. Effect of the Invention

[0009] According to the technology of the present disclosure, it is possible to improve the estimation accuracy of reference cross sections targeted at three-dimensional images. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of an image processing apparatus according to an embodiment. [Diagram 2]FIG. 4 is a flow diagram of a process executed by an image processing apparatus according to an embodiment. [Diagram 3] FIG. 1 illustrates a clinical definition of a reference cross section in one embodiment. [Figure 4] FIG. 4 is a diagram illustrating a reference cross section calculated by an image processing apparatus according to an embodiment. [Diagram 5] 5A and 5B are diagrams illustrating a correction parameter estimation process of the image processing apparatus according to the embodiment. [Figure 6] FIG. 4 is a flow diagram of a process executed by an image processing apparatus according to an embodiment. [Figure 7] FIG. 11 is another flowchart of the process executed by the image processing apparatus according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. Note that the present disclosure is not limited to the following embodiment, and can be modified as appropriate without departing from the gist of the present disclosure. In addition, in the drawings described below, parts having the same functions are given the same reference numerals, and their description may be omitted or simplified.

[0012] The image processing device according to the embodiment described below provides a function of estimating a predetermined reference cross section for a doctor or the like to observe (diagnose) an input three-dimensional image. The input image to be processed is a medical image, that is, an image of a subject (such as a human body) photographed or generated for the purpose of medical diagnosis, examination, research, etc., and is typically an image acquired by an imaging system called a modality. Examples of the input image include an ultrasound image obtained by an ultrasound diagnostic device, an X-ray CT image obtained by an X-ray CT device, and an MRI image obtained by an MRI device.

[0013] In the following description, a specific example of an image processing device will be described in detail, taking as an example a case where a transesophageal 3D ultrasound image of the mitral valve region of the heart is used as an input 3D image, and the position of an anatomical landmark of the mitral valve region, which is an observation target, is estimated. Note that using the mitral valve as an observation target is merely an example, and other parts may be observed.

[0014] First Embodiment The image processing device according to the first embodiment uses a three-dimensional image as an input image and estimates parameters (reference cross-sectional parameters) representing the position and orientation of a two-dimensional reference cross-section. In the estimation process, first, the reference cross-section parameters are estimated based on the input image. Hereinafter, the parameters estimated at this stage are referred to as "initial values ​​of the reference cross-section parameters" (initial parameters) as the initial values ​​of the parameters for obtaining the reference cross-section. Next, based on these initial parameters, three cross-sectional images, namely, plane A, plane B, and plane C, are cut out from the input image. Then, from each cross-sectional image, the correction amount of each different component in the reference cross-sectional parameters is estimated. In the following description, this is referred to as "correction amount of the reference cross-sectional parameters" (or simply correction amount). Finally, the correction amount estimated for each cross-sectional image is applied to the initial parameters to obtain the final reference cross-sectional parameters. Hereinafter, the parameters finally obtained in this manner are referred to as "final values ​​of the reference cross-sectional parameters" (final parameters).

[0015] Here, the reference cross section parameters are parameters that represent the position and posture of the reference cross section, as described above. The parameters are synonymous with parameters representing the position and orientation of a coordinate system (observation target coordinate system) representing the mitral valve (mitral valve) in the input three-dimensional image. In this embodiment, the position is expressed by three parameters representing the three-dimensional coordinates (cx, cy, cz) of the reference position (hereinafter also referred to as the center position) of the mitral valve described later. In addition, the orientation is expressed by a total of six parameters, including three parameters representing a long axis vector (lx, ly, lz) defined for the mitral valve as described later, and three parameters representing a short axis vector (sx, sy, sz). Note that the above-mentioned reference cross section parameters are merely exemplified as one embodiment of the technology of the present disclosure, and any expression form may be used as long as the information represents the position and orientation of the reference cross section. Details of the initial parameter estimation process of the reference cross section will be described in the explanation of step S203. In this embodiment, the position and orientation of each cross section of the A, B, and C planes in the observation target coordinate system are assumed to be known. Therefore, the position and orientation of each cross section image of the A, B, and C planes in the input three-dimensional image are uniquely calculated from the reference cross section parameters represented by the above nine parameters. Therefore, the cross-sectional images acquired by the cross-sectional image acquisition unit 43 are composed of a plurality of images whose positions and orientations have a predetermined relationship with each other.

[0016] Figures 3A to 3C show schematic diagrams of the clinical definition of each cross section. Figure 3A shows an example of plane A, which shows a cross section in which the aortic valve (301), anterior leaflet (302), and posterior leaflet (303) of the mitral valve can be observed simultaneously. Figure 3B shows an example of plane B, which shows a cross section that intersects with plane A at the boundary (304) between the anterior and posterior leaflets, and shows the mitral valve annulus (305 and 306) centered on this boundary. Figure 3C shows an example of plane C, which shows a cross section that cuts the mitral valve (307) and aortic valve (308) in a ring shape.

[0017] Based on the clinical definitions shown in Figs. 3A to 3C, Figs. 4A to 4E show the definitions of each cross section in this embodiment. Figs. 4A to 4C show the relationship of the positions and orientations of the A plane (402), the B plane (403), and the C plane (404) with respect to the input 3D image (401). The A plane, the B plane, and the C plane intersect at one point at the center position (407) and are perpendicular to each other. The long axis (405) is an axis that passes through the mitral valve from top to bottom as shown in Fig. 4D, and forms an in-plane vector on the A plane. The short axis (406), which is perpendicular to the long axis, is an axis that slices the mitral valve at the height of the anterior and posterior leaflets, and the axes 405 and 406 intersect at a point 407. The B plane is a cross section obtained by rotating the A plane by 90 degrees around the long axis (405). Plane C is a cross section perpendicular to both planes A and B, and axes 406 and 408 indicating in-plane vectors of plane C coincide with in-plane vectors passing through planes A and B from left to right, respectively.

[0018] As described above, the calculation of the correction amount of the reference cross-sectional parameters is performed by calculating the correction amount of different components of the reference cross-sectional parameters for each cross-sectional image. In the A plane, the in-plane component of the A plane of the center position (cx, cy, cz), i.e., how much the center position should be translated in the A plane, is calculated. In the B plane, the in-plane component of the long axis vector, i.e., how much the axis that penetrates the B plane vertically in the initial parameters should be rotated in the B plane, is calculated. And in the C plane, the in-plane component of the short axis vector, i.e., how much the axis that penetrates the C plane vertically in the initial parameters should be rotated in the C plane, is calculated. Details of the calculation process of the correction amount will be described in the explanation of step S205. Details of the process of reflecting the correction amount in the initial parameters to obtain the final parameters will be described in the explanation of step S206.

[0019] Note that, as long as the positions and orientations of three mutually orthogonal cross sections in a three-dimensional space can be expressed, the reference cross section parameters may be expressed in a format other than the above. For example, the orientation may be expressed with four parameters (three parameters of the rotation axis vector and one parameter of the rotation angle) using the Rodrigues rotation formula, which is a known technique. In this case, the reference cross section parameters will be a total of seven parameters, including the three parameters of the center position. Alternatively, the position and orientation can be expressed by a 4×4 rigid body transformation matrix. These expression formats can be converted into each other, so the processing of this embodiment can be executed in either case.

[0020] The configuration and processing of the image processing device of this embodiment will be described below. Fig. 1 is a block diagram showing an example of the configuration of an image processing system (also called a medical image processing system) including the image processing device of this embodiment. The image processing system 1 includes an image processing device 10 and a database 22. The image processing device 10 is connected to the database 22 via a network 21 in a communicable state. The network 21 includes, for example, a LAN (Local Area Network) or a WAN (Wide Area Network).

[0021] The database 22 holds and manages a plurality of images and information used in the processes described below. The information managed by the database 22 includes an input 3D image that is the subject of a reference cross section estimation process by the image processing device 10 and information on a learning model used in each estimation process. The image processing device 10 is capable of acquiring data held in the database 22 via the network 21. Note that the information on the learning model may be stored in an internal memory (ROM 32 or memory unit 34) of the image processing device 10 instead of in the database 22.

[0022] The image processing device 10 includes a communication IF (Interface) 31, a ROM (Read On The device includes a memory 32, a random access memory (RAM) 33, a storage unit 34, an operation unit 35, a display unit 36, and a control unit 37.

[0023] The communication IF 31 is a communication unit that is configured with a LAN card or the like and realizes communication between an external device (e.g., the database 22, etc.) and the image processing device 10. The ROM 32 is configured with a non-volatile memory or the like and stores various programs and various data. The RAM 33 is configured with a volatile memory or the like and is used as a work memory that temporarily stores programs and data being executed. The storage unit 34 is configured with a HDD (Hard Disk Drive) or the like and stores various programs and various data. The operation unit 35 is configured with a keyboard, mouse, touch panel, etc. and inputs instructions from a user (e.g., a doctor or a medical technician) to various devices. The display unit 36 ​​is configured with a display or the like and displays various information to the user.

[0024] The control unit 37 is configured with a CPU (Central Processing Unit) and the like, and controls the processing in the image processing device 10. The control unit 37 has, as its functional configuration, an image acquisition unit 41, an initial parameter estimation unit 42, a cross-sectional image acquisition unit 43, a learning model acquisition unit 44, a correction parameter estimation unit 45, a correction unit 46, and a display processing unit 51. The control unit 37 may also have a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), or the like.

[0025] The image acquisition unit 41 acquires an input three-dimensional image, which is a three-dimensional image of a subject to be input to the image processing device 10, from the database 22. The input three-dimensional image may be acquired directly from a modality. In that case, the image processing device 10 may be implemented in the console of the modality (imaging system). Details of the process will be described in detail in the explanation of step S201.

[0026] The initial parameter estimation unit 42 estimates initial parameters, which are initial values ​​of parameters for obtaining a reference cross section, by reducing the resolution of the input three-dimensional image acquired by the image acquisition unit 41. Details of the process will be described in detail in the description of step S203.

[0027] The learning model acquisition unit 44 acquires a learning model used by the initial parameter estimation unit 42 and the correction parameter estimation unit 45 from the database 22. Details of the process will be described in detail in the explanation of step S202.

[0028] The cross-sectional image acquiring unit 43 acquires at least one two-dimensional cross-sectional image from the three-dimensional image, using the input three-dimensional image acquired by the image acquiring unit 41 and the initial parameters of the reference cross section estimated by the initial parameter estimating unit 42. The cross-sectional image acquiring unit 43 also acquires at least one two-dimensional cross-sectional image from the three-dimensional image, using the final parameters corrected by the correcting unit 46. Details of the process will be described in detail in the description of steps S204 and S207.

[0029] The correction parameter estimation unit 45 is an index estimation unit that estimates the correction amount of the reference cross-sectional parameter for each cross-sectional image by using each cross-sectional image acquired by the cross-sectional image acquisition unit 43. Here, the correction amount of the reference cross-sectional parameter is an example of an index that contributes to the correction of the initial parameter. From the A plane, the correction amount of the center position is estimated, and from the B plane and the C plane, the correction amount of the direction of the axis (respectively called the long axis and the short axis) that penetrates each cross-section from top to bottom is estimated. The details of the process will be described in detail in the explanation of step S205.

[0030] The correction unit 46 calculates final parameters by reflecting the correction amounts of the reference cross section parameters estimated by the correction parameter estimation unit 45 on the initial parameters estimated by the initial parameter estimation unit 42. Details of the process will be described in detail in the description of step S206.

[0031] The display processing unit 51 displays the cross-sectional images obtained by cutting the input three-dimensional image with the reference cross-sectional parameters calculated by the correction unit 46 in an image display area of ​​the display unit 36 ​​in a display form that can be easily viewed by the user of the image processing device 10. Details of the process will be described in detail in the explanation of step S208.

[0032] Each component of the image processing device 10 functions according to a computer program. For example, the control unit 37 (CPU) reads and executes a computer program stored in the ROM 32 or the storage unit 34 using the RAM 33 as a work area, thereby realizing the function of each component. Note that some or all of the functions of the components of the image processing device 10 may be realized by using a dedicated circuit. Also, some of the functions of the components of the control unit 37 may be realized by using a cloud computer. For example, a calculation device located at a location different from the image processing device 10 may be communicably connected to the image processing device 10 via the network 21. Then, the image processing device 10 may transmit and receive data to and from the calculation device, thereby realizing the function of the components of the image processing device 10 or the control unit 37.

[0033] Next, an example of processing by the image processing device 10 in FIG. 1 will be described with reference to the flowchart in FIG.

[0034] (Step S201: Acquire input image) In step S201, the image processing device 10 receives an image acquisition instruction from the user via the operation unit 35. Then, the image acquisition unit 41 acquires an input three-dimensional image designated by the user from the database 22 and stores it in the RAM 33. Note that, other than acquiring the input three-dimensional image from the database 22, the input image may be acquired from ultrasound images captured by the ultrasound diagnostic device from time to time.

[0035] (Step S202: Acquire learning model) In step S202, the learning model acquisition unit 44 acquires from the database 22 two types of learning models used by the initial parameter estimation unit 42 and the correction parameter estimation unit 45 for their estimation, and stores them in the RAM 33.

[0036] The learning model used by the initial parameter estimation unit 42 represents a statistical tendency of the relationship between the input three-dimensional image and the reference cross section parameters which are the output values. Therefore, the learning model acquired by the initial parameter estimation unit 42 is an initial parameter estimation learning model created by a learning process using information including a pair of the learning three-dimensional image and the parameters for obtaining the reference cross section in the learning three-dimensional image.

[0037] On the other hand, the learning model used by the correction parameter estimation unit 45 is different for each cross-sectional image (A plane, B plane, C plane), and expresses the statistical tendency between the input cross-sectional image and the correction amount of the reference cross-sectional parameter in the cross-section. That is, three learning models are acquired: a learning model that estimates the correction amount of the in-plane component at the center position from the A plane image, a learning model that estimates the correction amount of the in-plane angle of the long axis vector from the B plane image, and a learning model that estimates the correction amount of the in-plane angle of the short axis vector from the C plane image. Therefore, the learning model acquired by the correction parameter estimation unit 45 is a learning model for correction parameter estimation created by a learning process using information including a pair of the learning cross-sectional image and an index that contributes to the correction of the reference cross-sectional parameter in the learning cross-sectional image. Details of the correction parameter estimation process for each cross-sectional image will be described in step S205.

[0038] In this embodiment, a convolutional neural network (CNN) is used as the learning model. In this case, a learning model for making each estimation is constructed in advance by learning a large number of case data in which input and output are paired as learning data. In this processing step, the learning model constructed in this manner is acquired. In addition to acquiring the previously constructed learning model from the database 22, a processing unit that performs the above-mentioned learning process may be configured in the control unit of the image processing device 10, and the learning in this step may be performed by the processing unit.

[0039] (Step S203: Estimation of initial parameters) In step S203, the initial parameter estimation unit 42 inputs volume data of the input three-dimensional image, and outputs initial parameters of the reference cross section consisting of the center position, major axis vector, and minor axis vector.

[0040] As described in step S202, in this embodiment, the initial parameters are estimated by using a convolutional neural network (CNN) that estimates the initial parameters from an input three-dimensional image. That is, a three-dimensional image is input to the CNN, and the reference cross-sectional parameters consisting of nine parameters obtained as the output of the CNN are obtained as the initial parameters.

[0041] Here, the three-dimensional image input to the CNN is not the same as the input three-dimensional image acquired in step S201, but is a "coarse" image with reduced resolution. For example, the input three-dimensional image is a volume image in which the length of each voxel is 0.6 mm and 256 x 256 x 256 voxels, that is, a range of 153.6 mm on each side, are expressed. In this step, the number of voxels on each side of the input three-dimensional image is reduced to 1 / 8, to 32 x 32 x 32 voxels. That is, the range to be expressed remains unchanged at 153.6 mm on each side, and the image is converted to one with a length of 4.8 mm per voxel. In this way, the calculation time and memory usage required for inference using the CNN can be reduced compared to the case where the input three-dimensional image is used as is. In addition, known image processing such as pixel value normalization using the pixel value mean and variance of the input three-dimensional image and contrast correction may be further applied. Note that any known method may be used for the above-mentioned resolution conversion processing. For example, voxel values ​​can be sampled at steps corresponding to the reduction width, or the average pixel value of voxels in a range corresponding to the reduction width can be used. Furthermore, the image processing such as the above-mentioned resolution conversion processing and pixel value normalization can be performed in any order.

[0042] In this embodiment, the reference cross section is expressed by nine parameters, and in the processing of this step, an example is described in which the above nine values ​​are directly estimated as the output of the CNN estimation, but other expression formats may be used for the output of the CNN estimation. For example, the reference cross section parameters may be expressed by seven parameters (three parameters for the center position, three parameters for the rotation axis vector, and one parameter for the rotation angle), and the output of the CNN estimation may be the output of these seven parameters. In this case, after the estimation by CNN, it may be converted into an expression format of the above nine parameters. Similarly, the output of the CNN estimation may be a 4 x 4 rigid body transformation matrix. It may also be something that

[0043] In this embodiment, the case where the A, B, and C planes are mutually orthogonal is described as an example, but as long as the position and orientation relationship between each cross section is defined in advance, the A, B, and C planes do not necessarily have to be mutually orthogonal. For example, the B plane may be defined as a cross section rotated 80 degrees around the long axis, not orthogonal to the A plane. Even in this case, it is possible to perform the processing of this step in the same way as in the case where the three cross sections are orthogonal. Also, if the position and orientation of each cross section in the observation target coordinate system are known, the intersection position of the three cross sections does not necessarily have to coincide with the center position of the reference cross section parameters (= the reference position of the observation target). Similarly, the three cross sections do not necessarily have to share the in-plane axis, and the image center of each reference cross section image does not necessarily have to coincide with the center position of the reference cross section parameters or the intersection position of the three cross sections.

[0044] (Step S204: Acquiring a cross-sectional image) In step S204, the cross-sectional image acquisition unit 43 cuts out a two-dimensional cross-sectional image using the input three-dimensional image and the initial parameters estimated in step S203. In this embodiment, when acquiring each two-dimensional cross-sectional image, the image is cut out so that the center position of the initial parameters of the reference cross section is at the center of the cross-sectional image. That is, even if the center position (407) is not the center of the input three-dimensional image (401) as shown in Figures 4A to 4C, the center of each cross-sectional image of the A plane (4021), the B plane (4031), and the C plane (4041) is the center position (407) as shown in Figures 4D to 4F.

[0045] Here, the spatial resolution of the two-dimensional cross-sectional image is higher than the resolution of the three-dimensional image after the resolution reduction used in step S203. For example, the two-dimensional cross-sectional image is an image of 128×128 pixels with a length per pixel (pixel size) of 1.2 mm. In this case, compared with the image used in step S203 with a length per voxel of 4.8 mm, the image has four times the resolution per side. In this way, a high-resolution image can be used for estimating the reference cross-sectional parameters without significantly increasing the amount of calculation or memory usage.

[0046] Here, the cross-sectional image acquisition process will be described in more detail with reference to FIG. 4A to FIG. 4F. When acquiring an A-plane cross-sectional image (4021), first, the input three-dimensional image (401) is sampled with the center position 407, the vertical vector as axis 405, and the horizontal vector as axis 406. Here, the length per pixel is set to be the same as that of the input three-dimensional image (0.6 mm), and a range of 256×256 pixels (153.6 mm×153.6 mm) is sampled. At this time, if the sampling position exceeds the defined range of the input three-dimensional image, the pixel value of that position is set to 0. Next, the resolution of the thus obtained two-dimensional image of 256×256 pixels is reduced to 1 / 2 to generate a two-dimensional image of 128×128 pixels. That is, a two-dimensional image with a length per pixel of 1.2 mm is obtained while the defined range remains unchanged at 153.6 mm×153.6 mm. In this manner, the A-plane cross-sectional image can be acquired. The B plane cross-sectional image (4031) is calculated in the same procedure as the A plane cross-sectional image, with the up-down vector as axis 405 and the left-right vector as axis 408. The C plane cross-sectional image (4041) is also calculated in the same procedure as the A and B plane cross-sectional images, except that the up-down vector is axis 406 and the left-right vector is axis 408.

[0047] (Step S205: Estimation of correction parameters) In step S205, the correction parameter estimation unit 45 receives each cross-sectional image calculated in step S204 as an input, and calculates the amount of correction of the components of the reference cross-sectional parameters corresponding to each type of cross-section using the learning model. In this way, the correction parameter estimation unit 45 estimates an index that contributes to the correction of the initial parameters in the image plane of the cross-sectional image. Here, The correction amount estimation for the A, B, and C cross-sectional images is performed independently. The estimation for each cross-sectional image is performed using CNN that uses the cross-sectional images as input.

[0048] Here, the estimation process of the correction parameters, which are the correction amounts for displacing the reference cross section obtained by the initial parameters, will be specifically described. The estimation process of the correction parameters using the A-plane cross-sectional image will be described with reference to FIG. 5A. In the A-plane cross-sectional image (501) acquired based on the initial parameters, the center position (507a) of the reference cross section is located at the center of the A-plane cross-sectional image. At this time, in this step, the displacement amount for moving the position of the reference cross section obtained by the initial parameters to the target position of the reference cross section, that is, "how much should this center position be moved in the A-plane?", is estimated. That is, it is estimated how much the center position estimated as the initial parameter in step S203 is shifted from the center position that is the target position of the reference cross section determined by a predetermined standard (for example, anatomical meaning or clinical guideline). Note that instead of estimating the amount of movement of the center position, the corrected center position (two-dimensional coordinate value in the A-plane) may be estimated. In this case, for example, a position different from the original center position, such as point 517 in FIG. 5A, is estimated as the corrected center position.

[0049] The estimation process of the correction parameters using the B plane cross-sectional image will be described with reference to FIG. 5B. In the initial parameters, the long axis vector (505b) is an axis that passes through the B plane cross-sectional image from top to bottom. At this time, in this step, the amount of rotation for rotating the posture of the reference cross section obtained by the initial parameters to the target posture of the reference cross section, that is, "how much should this long axis vector be rotated in the B plane?", is estimated. That is, it is estimated how much the long axis vector estimated as the initial parameter in step S203 deviates from the long axis vector that is the target of the reference cross section determined by a predetermined standard (anatomical meaning or clinical guideline). Note that instead of estimating the amount of rotation of the long axis vector, the direction of the long axis vector after correction (two-dimensional vector in the B plane) may be estimated. In this case, for example, a vector with a different direction from the original long axis vector, such as vector 515 in FIG. 5B, is estimated as the long axis vector after correction.

[0050] Alternatively, instead of directly estimating the corrected long axis vector, the corrected long axis vector may be calculated indirectly from the estimation results of other indices. For example, the coordinate values ​​of two points (points 510 and 511 in FIG. 5B) where the B plane intersects with the mitral annulus may be estimated, and a vector perpendicular to the vector connecting the two points may be calculated as the corrected long axis vector. Alternatively, the coordinate values ​​of two points (not shown) through which the long axis should pass other than these two points may be estimated, and a vector connecting the estimated two points may be calculated as the corrected long axis vector.

[0051] The process of estimating correction parameters using the C plane cross-sectional image will be described with reference to FIG. 5C. In the initial parameters, the short axis vector (506c) is an axis that passes through the C plane cross-sectional image from top to bottom. At this time, in this step, the amount of rotation for rotating the posture of the reference cross section obtained by the initial parameters to the target posture of the reference cross section, that is, "how much should this short axis vector be rotated within the C plane", is estimated. That is, it is estimated how much the short axis vector estimated as the initial parameter in step S203 deviates from the target short axis vector of the reference cross section determined by a predetermined standard (anatomical meaning or clinical guideline).

[0052] As in the case of plane B, instead of estimating the amount of rotation of the minor axis vector, the direction of the minor axis vector after correction (two-dimensional vector in plane C) may be estimated. In this case, for example, a vector with a different direction from the original minor axis vector, such as vector 516 in Fig. 5C, is estimated as the minor axis vector after correction.

[0053] Alternatively, instead of directly estimating the corrected short-axis vector, the corrected short-axis vector may be calculated indirectly from the estimation results of other indices. For example, the coordinate values ​​of two points (points 512 and 513 in FIG. 5C) where the B plane intersects with the mitral annulus may be estimated, and a vector perpendicular to the vector connecting the two points may be calculated as the corrected short-axis vector. Alternatively, the coordinate values ​​of two points (not shown) through which the short axis should pass other than these two points may be estimated, and a vector connecting the estimated two points may be calculated as the corrected short-axis vector.

[0054] Also, the corrected short axis vector may be calculated by image processing without using a learning model. For example, a vector in the direction of the maximum diameter of the annulus depicted in the C plane cross-sectional image may be calculated, and a vector perpendicular to the vector may be used as the short axis vector. Similarly, in the estimation process of the correction parameters using the A plane cross-sectional image and the B plane cross-sectional image, the feature point positions required for correction may be estimated by image processing such as template matching, and the correction may be performed using the estimated feature points.

[0055] In this embodiment, all three cross sections, namely, plane A, plane B, and plane C, are used as the calculation target of the correction amount, and when the cross section image acquisition unit 43 acquires two or more cross section images, the index estimated by the correction parameter estimation unit 45 differs for each cross section image. However, it is not necessary to calculate the correction amount for all three cross sections. For example, only one cross section may be used as the calculation target, or two cross sections, plane A and plane B, may be used as the calculation target. In this case, for components of the reference cross section parameters for which the correction amount has not been calculated, the estimated values ​​as initial parameters may be used as final parameters. Alternatively, the correction amounts for two or more components may be calculated from one cross section image, such as calculating the correction amounts for both the center position and the long axis vector using the A cross section image.

[0056] In addition, in the present embodiment, the case where a two-dimensional cross-sectional image is used in the estimation of the correction parameters in step S205 has been described as an example, but a three-dimensional image consisting of a small number of slices may be used. In this case, the cross-sectional image acquisition unit 43 acquires a cross-sectional image including a number of slices corresponding to a predetermined thickness in the three-dimensional image from the three-dimensional image based on the initial parameters. For example, the input of the CNN may be a three-dimensional image of a small number of slices (for example, two slices before and after the output cross-sectional image as the center), and the output may be an in-plane component correction amount as in the above-mentioned processing. In this way, it is possible to perform correction amount estimation taking into account the image characteristics of the nearby three-dimensional space. Alternatively, the output may not be limited to the in-plane component, and the correction amount in the three-dimensional space may be estimated. That is, for example, the correction amount of the three-dimensional coordinates of the center position is calculated from a three-dimensional image consisting of slices around the A-plane cross-sectional image. In this way, it is possible to perform correction amount calculation with higher accuracy.

[0057] (Step S206: Correction of initial parameters) In step S206, the correction unit 46 corrects the initial parameters using the initial parameters estimated in step S203 and the correction amount estimated in step S205, and calculates final parameters.

[0058] The correction of the initial parameters is performed according to the following procedure. First, the correction amount (movement amount and rotation amount) defined in the two-dimensional space of each cross section is converted into the movement amount and rotation amount in the three-dimensional space. Next, the initial parameters of the reference cross section are corrected using the converted movement amount and rotation amount to calculate the final parameters. That is, in the previous step S205, the correction amount estimated independently for each component for each cross section image is simultaneously reflected in this step. Note that, if the center position and axis vector after correction are estimated in step S205 instead of the correction amount, the center position and axis vector obtained by converting the estimated value into the three-dimensional space coordinate system, which is the coordinate system of the input image, are set as the values ​​of the final parameters.

[0059] In addition, in the case where the correction amount for the same component of the parameter is calculated for a plurality of cross sections in step S205, the correction amount is selected and combined in this step. For example, when the A-plane movement amount of the central position is estimated from the A-plane cross section, and the B-plane movement amount of the central position and the B-plane rotation amount of the long axis vector are estimated from the B-plane cross section, two types of correction amounts, the A-plane movement amount and the B-plane movement amount, are present for the central position. At this time, for the vertical movement amount, which is a common component in the two movement amounts, a value obtained by integrating both of them (for example, a simple average value or a weighted average value) is used. In this way, it is possible to reflect the two types of correction amounts in a well-balanced manner. In addition, the reliability of the correction parameter calculation process may be evaluated, and the correction amount to be adopted may be selected based on the reliability. For example, when the correction amount is greater than a predetermined value (for example, the movement amount of the central position is 10 mm) or more, only the correction result whose absolute value of the movement amount is less than a predetermined value may be adopted for correction based on the intention of regarding the estimation of the correction amount as having failed. In this way, it is possible to reduce the risk of outputting a correction result with low reliability far from the initial parameters. Alternatively, the correction amount may be weighted based on a certain criterion. For example, weighting may be performed based on the contrast of the cross-sectional image so that the correction amount by a cross-sectional image with high contrast is more important. This makes it possible to calculate a correction amount that places emphasis on a more reliable correction amount.

[0060] (Step S207: Acquiring a cross-sectional image) In step S207, the cross-sectional image acquisition unit 43 extracts a two-dimensional cross-sectional image by using the input three-dimensional image and the corrected reference cross-sectional parameters estimated in step S206. The two-dimensional cross-sectional image acquired in this way is a cross-sectional image in which the center position of the reference cross-section and each axis vector are corrected as described above.

[0061] (Step S208: Display of estimation results) In step S208, the display processing unit 51 displays, in an easily viewable display form, the cross-sectional images obtained by cutting the input three-dimensional image using the final parameters in step S207 within the image display area of ​​the display unit .

[0062] In addition, instead of displaying based on the final parameters, the display may be performed in a display form that combines the "sectional image based on the initial parameters" acquired in step S204 and the "correction amount" calculated in step S205. For example, the sectional images are generated based on the initial parameters of the reference cross section. Then, information that visualizes the amount of movement of the center position and the amount of rotation of the vector with arrows or the like is superimposed on these sectional images. In this case, it is possible to omit the correction process in step S206. In addition, when the purpose is analysis or measurement based on the reference cross section, the display process in step S208 is not essential, and each sectional image may be generated from the input 3D image using the final parameters obtained in step S206 and stored in a storage device or output to the outside. In addition, the final parameters obtained in step S206 may be output to the storage unit 34 or to the outside.

[0063] According to this embodiment, in the process of calculating the reference cross section parameters from the three-dimensional image, the initial parameters estimated from the three-dimensional image with low resolution are corrected based on the two-dimensional cross section image, which enables highly accurate estimation without increasing the amount of calculation.

[0064] <Modification 1 of the First Embodiment> Hereinafter, a modified example of the first embodiment will be described. In the first embodiment, an example was shown in which the processing target was a 3D ultrasound image of the mitral valve region of the heart, but the technology of the present disclosure can also be implemented in cases where an image of a region of the heart other than the mitral valve, an image of an organ other than the heart, or an image by another modality is used.

[0065] An example of the application of the present invention to images of regions other than the mitral valve region of the heart captured by a modality other than an ultrasound image is the detection of specific structures such as the diencephalon and corpus callosum from brain MRI images. The axial plane can be used as the reference plane. In this case, the initial value of this reference plane (axial plane) is first calculated using CNN. Then, images of the axial plane and the coronal and sagittal planes perpendicular to the axial plane are acquired. Furthermore, point coordinates indicating the positions of specific anatomical landmarks representing the brain structure on the images are estimated. Finally, the estimated point coordinates of these landmarks are used to correct the parameters representing the axial plane.

[0066] In this way, according to this modification, the technology of the present disclosure can be applied to images of modalities other than 3D ultrasound images and to regions other than the mitral valve of the heart.

[0067] <Modification 2 of the First Embodiment> Next, another modified example of the first embodiment will be described. In the first embodiment, when estimating the initial values ​​of the reference cross section parameters in step S203, the "relationship between the image and the parameter value" is learned using CNN, but the technology of the present disclosure can be implemented using a method other than learning the relationship between the image and the parameter in this manner.

[0068] One example is a method of using a known classifier such as Random Forests to classify whether an input image is appropriate as a reference cross section. In this case, in the learning stage, a learning 3D image is first cut out with a large number of random cross section parameters. At this time, an image cut out with parameters appropriate (close to the reference cross section) as a reference cross section is labeled as a "reference cross section", and an image cut out with parameters inappropriate (far from the reference cross section) as a reference cross section is labeled as a "not a reference cross section". Here, "close" or "far" from the reference cross section refers to a case where the difference in position and orientation from the reference cross section is less than or greater than a predetermined value (e.g., 5.0 mm or 5.0 degrees). These labeled images are then input to Random Forests to train a classifier. In the estimation stage, a large number of random cross section parameters are generated, and a large number of cross section images are cut out from the input 3D image using these parameters, and each cross section image is classified as "reference cross section or not" using a classifier. Then, the average value of the parameters used to generate the cross section images determined to be "reference cross sections" is output as the estimation result of the reference cross section parameters.

[0069] Alternatively, it is possible to estimate the initial parameters by applying an algorithm that is not based on learning data. For example, a two-dimensional template representing a typical aspect of each cross-sectional image (A plane, B plane, C plane) of the reference cross section may be prepared, and template matching may be performed to search for a position and orientation to which the template fits well in the input three-dimensional image. Furthermore, the image processing device 10 may not automatically estimate the initial parameters, but may accept input of the initial values ​​of the reference cross-sectional parameters by the user via the operation unit 35.

[0070] In this way, according to this modification, it is possible to determine the initial values ​​of the reference cross section parameters based on various methods, not limited to CNN.

[0071] <Modification 3 of the First Embodiment> In the first embodiment, the position and orientation of three cross sections (A, B, and C planes) in the observation target coordinate system are known, and the position and orientation of each reference cross section is uniquely calculated from common reference cross section parameters. However, the technology disclosed herein can be applied even when each reference cross section is defined independently.

[0072] As an example, a case will be described in which the three cross sections of planes A, B, and C do not share a common center position and are non-orthogonal. In the first embodiment, the three cross sections are expressed using nine parameters (three position parameters, three major axis vector parameters, and three minor axis vector parameters), but when the three cross sections are independent of each other, nine parameters are required to express each cross section. In other words, the reference cross section parameters in this modified example represent the position and posture of the observation object. These are not parameters that are used to estimate the reference cross sections, but parameters that directly represent the position and orientation of each of the reference cross sections. Therefore, in step S203, a total of 27 parameters are estimated as the initial parameters of the reference cross sections. In addition, in step S205, as in the first embodiment, the correction amount of the in-plane component of each cross section is calculated for each cross section, but if the cross sections are independent of each other, the correction amount calculated using a certain cross section does not affect the other cross sections. In other words, the correction amount estimated for each cross section is applied independently to each cross section.

[0073] In addition, each cross section may be completely independent as described above, or some cross sections may share some parameters and only some parameters may be independent, such as sharing the center position and the long axis but only the angle at which the cross sections intersect with each other being indefinite. In such a case, only the indefinite parameters are estimated as initial parameters. In this case, the correction amount calculated using a certain cross section is used to correct the parameters of the cross section only when the correction amount affects the parameters shared with other cross sections.

[0074] Alternatively, the technology of the present disclosure is applicable in the same manner as in the first embodiment, not limited to the case where three reference cross sections, namely, planes A, B, and C, are used, but also when the number of reference cross sections is one or two, or four or more, as long as the reference cross sections can be expressed by a predetermined number of parameters.

[0075] <Second embodiment> Next, a second embodiment will be described. In the following description, the same components and processes as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted. The configuration of an image processing device 10 according to this embodiment is the configuration shown in FIG.

[0076] The image processing device 10 according to the second embodiment is a device that inputs a 3D ultrasound image of a cardiac region and estimates a reference cross section for observation, as in the first embodiment. In the first embodiment, the correction amount estimated in step S205 is unconditionally reflected in the final estimation result. However, as in the present embodiment, a determination may be made as to whether or not to reflect the estimated correction amount in the final result based on a certain criterion. In this way, when there are multiple indices estimated by the correction parameter estimation unit 45, the correction unit 46 can determine for each index whether or not to correct the initial parameters based on the index.

[0077] As described below, in the image processing device 10 of this embodiment, the processing of the correction unit 46 and the display processing unit 51 differs from the processing in the first embodiment.

[0078] The correction unit 46 applies the correction amount of the reference cross section parameters estimated by the correction parameter estimation unit 45 to the initial parameters estimated by the initial parameter estimation unit 42 to calculate the final parameters. At this time, the correction unit 46 acquires the user's judgment result on whether or not to accept the correction result through the operation unit 35 and decides whether or not to apply the correction amount of the reference cross section parameters. Then, when it is decided to apply the correction amount, the correction amount of the reference cross section parameters is applied to the initial parameters.

[0079] The display processing unit 51 has a function of displaying a cross-sectional image based on the reference cross-sectional parameters in a display form that allows the user to determine whether or not to accept the correction result of the reference cross-sectional parameters, in addition to the functions of the display processing unit 51 described in the first embodiment. Details of the processing by the correction unit 46 and the display processing unit 51 will be described later in the description of step S406.

[0080] Next, an example of the processing of the image processing device 10 in the second embodiment will be described with reference to the flowchart in FIG. 6. Here, the processing of steps S601 to S605, S608, and S609 is the same as the processing of steps S201 to S205, S207, and S208 in the flowchart (FIG. 2) of the first embodiment. Below, processing steps that differ from the first embodiment will be described. This article will only explain the top.

[0081] (Step S606: Determine correction parameters) In step 606, the correction unit 46 determines whether or not to adopt the correction amount of the reference cross section parameters estimated in step S605 for each component of the reference cross section parameters by controlling the operation unit 35 and the display processing unit 51. The processing procedure will be specifically described below.

[0082] First, the display processing unit 51 displays images necessary for the user to judge the judgment result. For example, a cross-sectional image generated based on the reference cross-sectional parameters before correction (initial parameters) and a cross-sectional image generated based on the reference cross-sectional parameters reflecting the correction amount are displayed side by side. The user can judge whether the correction amount is appropriate by comparing the cross-sectional image generated based on the initial parameters with the cross-sectional image generated with the parameters reflecting the correction amount.

[0083] If the user can determine the appropriateness of the correction amount, a display method other than the above may be used. For example, an arrow or the like indicating the correction amount (amount of movement) of the center position or the correction amount (amount of rotation) of the axis vector may be superimposed on a cross-sectional image generated based on the initial parameters before correction. Alternatively, as shown in Figures 5A to 5C, the center positions and axis vectors before and after correction may be displayed as points or lines.

[0084] Secondly, the correction unit 46 acquires the selection result of "accept or not accept the correction result" by the user through the operation unit 35. At this time, the correction unit 46 receives the selection result of whether or not to accept the correction result for each component of the reference cross section parameters. That is, the correction unit 46 judges whether or not to accept the correction result for each of the three types of components of the center position, the major axis vector, and the minor axis vector.

[0085] The correction unit 46 may accept not only the selection result of whether to accept the correction result or not, but also the input of the correction result by the user, and correct the initial parameters based on the accepted correction result. For example, the user may operate the operation unit 35 while checking the cross-sectional image after correction, and interactively correct the center position and the attitude (specifically, the value of the axis vector) on the cross-sectional image. In this case, the correction unit 46 adopts the correction instruction from the user as a new correction amount. Also, the instruction of whether to accept the correction result obtained in this way is stored in the RAM 33 together with the correction amount. Alternatively, the user may always manually perform correction based on the initial parameters without selecting whether to accept the correction result or not.

[0086] Note that the user may not manually determine whether to accept the correction result, but may automatically determine based on a predetermined criterion. For example, if the correction amount calculated in step S605 (i.e., the amount of movement of the center position or the amount of rotation of the axis vector) is greater than a predetermined value, the correction unit 46 may determine that the estimation of the correction amount has failed and may determine not to accept the correction result. Alternatively, the correction unit 46 may obtain a classifier that quantitatively evaluates the "appropriateness (likelihood) of each cross-sectional image" for each cross-sectional image, and may determine that the estimation of the correction amount has failed if the likelihood of the result of classifying the cross-sectional image after correction with the classifier falls below a predetermined value. Also, for example, if the correction unit 46 determines that the estimation of the correction amount has failed based on these automatic determination results, it may display a message on the display unit 36 ​​to prompt the user to correct the correction amount.

[0087] (Step S607: Correction of initial parameters) In step S607, the correction unit 46 corrects the initial parameters using the initial parameters estimated in step S603 and the correction amount estimated in step S605, and calculates the final parameters. This is the same as the process of step S206 in the first embodiment, except that the above is used. That is, the correction unit 46 reflects only the parameters of the components determined to "accept the correction result" in step S606 in the correction of the initial parameters.

[0088] As described above, according to the present embodiment, by using the judgment of the user for correcting the initial parameters, it is possible to perform more flexible and high-quality estimation of the reference cross section parameters.

[0089] <Third embodiment> Next, a third embodiment will be described. In the following description, the same configurations and processes as those in the first and second embodiments are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0090] The image processing device according to the third embodiment is a device that estimates a reference cross section of an input three-dimensional image, similarly to the first and second embodiments. The configuration of an image processing device 10 according to this embodiment is the configuration shown in FIG.

[0091] Moreover, the image processing device 10 of this embodiment calculates the initial parameters of the reference cross section, and calculates the correction amount using the cross section image generated based on the initial parameters, as in the first and second embodiments. However, in the first and second embodiments, the correction amounts of three types of components, namely the center position, the long axis vector, and the short axis vector, are estimated independently (in parallel), and these correction amounts are simultaneously reflected in the final parameters. However, in this embodiment, the estimation of the correction amount of each component and the reflection of the correction amount in the reference cross section parameters are performed sequentially. That is, in the image processing device 10 of this embodiment, the order of the components to be corrected is predetermined. Information regarding the order of the components to be corrected may be stored in the storage unit 34 or the like.

[0092] Then, first, the correction parameter estimation unit 45 estimates the amount of correction for the first component, and the cross-sectional image acquisition unit 43 acquires a cross-sectional image (corrected cross-sectional image) cut out by the reference cross-sectional parameters corrected based on the estimated first component. Then, the correction parameter estimation unit 45 estimates the amount of correction for the second component based on the corrected cross-sectional image. Similarly, the cross-sectional image acquisition unit 43 acquires a corrected cross-sectional image corrected based on the estimation result, and the correction parameter estimation unit 45 estimates the amount of correction for the third component based on the corrected cross-sectional image.

[0093] As in the first embodiment, the cross-sectional image acquisition unit 43 generates a two-dimensional cross-sectional image by processing the input three-dimensional image acquired by the image acquisition unit 61 based on the reference cross-sectional parameters. In addition, the cross-sectional image acquisition unit 43 not only generates cross-sectional images based on the initial parameters of the reference cross-section estimated by the initial parameter estimation unit 42, but also generates cross-sectional images based on the reference cross-sectional parameters corrected by the correction unit 46.

[0094] The correction parameter estimation unit 45 calculates the correction amount of the component associated with the type of each cross-sectional image based on the two-dimensional cross-sectional image generated by the cross-sectional image acquisition unit 43, similarly to the first embodiment. However, unlike the first embodiment, the correction parameter estimation unit 45 does not simultaneously and independently calculate the correction amount for all the A, B, and C planes, but only estimates the correction amount for the currently targeted cross-section based on a predetermined order. For example, when it is determined that "correction is performed in the order of A, B, and C planes", the correction parameter estimation unit 45 estimates the correction amount using the cross-sectional image of the B plane after estimating the correction amount for the A plane and reflecting the correction amount in the reference cross-sectional parameters. Furthermore, the correction parameter estimation unit 45 estimates the correction amount using the cross-sectional image of the C plane after estimating the correction amount for the B plane and reflecting the correction amount in the reference cross-sectional parameters. The specific order of processing will be described later.

[0095] As in the first embodiment, the correction unit 46 corrects the reference cross section parameters using the correction amounts estimated by the correction parameter estimation unit 45. In the first embodiment, the correction unit 46 corrects the reference cross section parameters using all the correction amounts of the A, B, and C planes. On the other hand, in the present embodiment, the correction unit 46 corrects the reference cross section parameters using only the correction amounts of the components estimated by the correction parameter estimation unit 45.

[0096] Next, an example of the processing in the image processing device 10 according to the present embodiment will be described with reference to the flowchart in Fig. 7. Note that the processing from S701 to S703, S708, and S709 is the same as the processing from S201 to S203, S207, and S208 in the flowchart in Fig. 2, respectively, and therefore detailed description thereof will be omitted here.

[0097] (Step S704: Acquiring a cross-sectional image) In step S704, the cross-sectional image acquiring unit 43 generates a two-dimensional cross-sectional image by cutting the input three-dimensional image acquired by the image acquiring unit 41 based on the reference cross-sectional parameters.

[0098] In step S204 in the first embodiment, a two-dimensional cross-sectional image is always calculated based on the initial parameters estimated in step S203. However, in this embodiment, when the process returns to step S704 after the subsequent steps S705 to S707, a two-dimensional cross-sectional image is acquired in step S704 based on the reference cross-sectional parameters corrected in step S706.

[0099] For example, when the process returns to step S704 after the reference cross section parameters are corrected using plane A in step S706, the cross section image acquired in S704 is a cross section image in which only the component corresponding to plane A, i.e., the center position, is corrected. In this case, in step S704, the center position after correction is adopted, and the other components (long axis vector and short axis vector) are adopted as initial parameters, and the cross section image is acquired by the cross section image acquisition unit 43.

[0100] (Step S705: Estimation of correction parameters) In step S705, the correction parameter estimation unit 45 estimates the correction amount of the component of the reference cross section parameter corresponding to the type of cross section, using the cross section image acquired in step S704.

[0101] In step S705, unlike step S205 in the first embodiment, correction parameters for one type of cross-sectional image are estimated in a predetermined order. For example, if the order of correction is set to plane A, plane B, and plane C, and step S705 is performed after the correction of plane A is completed, the parameter correction amount is estimated for the plane B cross-sectional image in step S705. The process of estimating the correction amount for each cross-sectional image is the same as the process of step S205 in the first embodiment.

[0102] (Step S706: Correcting parameters) In step S706, the correction unit 46 calculates the corrected reference cross section parameters using the correction amounts of the parameters estimated in step S705.

[0103] In step S206 in the first embodiment, the initial parameters are input, and the correction amounts of the center position, major axis vector, and minor axis vector are all simultaneously reflected in the initial parameters. On the other hand, in the process of step S705 in this embodiment, the initial parameters are corrected for the components for which the correction amounts of the parameters obtained by the processes of steps S704 to S707 have been estimated. In other words, some components of the reference cross section parameters are updated. Note that the process itself of correcting the reference cross section parameters using the estimated correction amounts is the same as the process of the first embodiment. It is.

[0104] (Step S707: Determine whether correction of all parameters is complete) In step S707, the correction unit 46 judges whether or not the correction of all components of the reference cross-sectional parameters to be corrected has been completed. For example, when the order of correction is set to planes A, B, and C, the correction unit 46 considers the correction of the initial parameters to be completed when the correction of each cross-section has been completed (S707: YES), and proceeds to step S708. On the other hand, when there is a cross-section for which the correction has not been completed, that is, when some components of the initial parameters remain as reference cross-sectional parameters, the correction unit 46 considers the correction to be incomplete (S707: NO), and returns the process to step S704.

[0105] According to this embodiment, the correction amount of the remaining components (e.g., long axis vector) of the reference cross section parameters is estimated based on a cross section image reflecting the correction of a certain component (e.g., center position). That is, the image processing device 10 according to this embodiment sequentially executes the correction of the reference cross section parameters. Therefore, according to this embodiment, it is possible to reduce the possibility of incompatibility or contradiction occurring between the correction results of each component of the reference cross section parameters, compared to the case where the correction amount of each component of the reference cross section parameters is independently estimated and reflected simultaneously. This makes it possible to calculate higher quality reference cross section parameters.

[0106] In this embodiment, the correction amount is reflected in the order of the A plane, the B plane, and the C plane, but a different order may be used. Also, correction may not be performed for any one of the cross sections. Alternatively, in combination with the process described in the second embodiment, it may be determined for each cross section whether or not to reflect the correction amount. In this way, the correction parameter estimation unit 45 can estimate an index based on a part of multiple images constituting the cross-sectional image, and sequentially switch in a predetermined order which image of the multiple images the index is to be estimated based on.

[0107] Although the above is a description of the present embodiment, the configuration and processing of the electronic device are not limited to the above embodiment, and various modifications are possible within the scope of the technical idea and identity of the present invention. For example, the processing of each of the above embodiments can be appropriately combined and implemented to improve the estimation accuracy of the reference cross section parameters.

[0108] <Other embodiments> The technology of the present disclosure can be embodied as, for example, a system, an apparatus, a method, a program, or a recording medium (storage medium), etc. Specifically, the technology may be applied to a system composed of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to an apparatus composed of a single device.

[0109] Needless to say, the object of the present invention can be achieved by the following: That is, a recording medium (or storage medium) on which is recorded a program code (computer program) of software that realizes the functions of the above-mentioned embodiments is supplied to a system or device. The storage medium is, of course, a computer-readable storage medium. Then, a computer (or a CPU or MPU) of the system or device reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-mentioned embodiments, and the recording medium on which the program code is recorded constitutes the present invention.

[0110] The disclosure of this embodiment includes the following configuration. (Configuration 1) an image acquisition unit for acquiring a three-dimensional image of a subject; an initial parameter estimation unit that estimates initial parameters, which are initial values ​​of parameters for obtaining a reference cross section, by reducing the resolution of the three-dimensional image; a cross-sectional image acquisition unit that acquires at least one cross-sectional image from the three-dimensional image based on the initial parameters; an index estimation unit that estimates an index that contributes to correction of the initial parameters for each of the cross-sectional images; An image processing device comprising: (Configuration 2) 2. The image processing device according to claim 1, wherein the cross-sectional image acquisition unit acquires a cross-sectional image including a plurality of slices corresponding to a predetermined thickness in the three-dimensional image from the three-dimensional image based on the initial parameters. (Configuration 3) 3. The image processing device according to configuration 1 or 2, wherein a resolution of the cross-sectional image acquired by the cross-sectional image acquisition unit is higher than a resolution of the three-dimensional image after resolution reduction. (Configuration 4) 4. The image processing device according to any one of configurations 1 to 3, wherein when the cross-sectional image acquisition unit acquires two or more cross-sectional images, the indices estimated by the index estimation unit differ for each cross-sectional image. (Configuration 5) 5. The image processing device according to any one of claims 1 to 4, wherein the initial parameter estimation unit estimates the initial parameters based on an image obtained by reducing the resolution of the three-dimensional image. (Configuration 6) Further comprising a learning model acquisition unit for acquiring a learning model, the learning model is at least one of an initial parameter estimation learning model created by a learning process using information including a pair of a learning 3D image and a parameter for obtaining the reference cross section in the learning 3D image, and an index estimation learning model created by a learning process using information including a pair of a learning cross section image and an index contributing to correction of the reference cross section parameters in the learning cross section image; At least one of the initial parameter estimation unit and the index estimation unit performs the estimation based on the learning model. 6. The image processing device according to any one of configurations 1 to 5. (Configuration 7) 7. The image processing device according to any one of configurations 1 to 6, wherein the initial parameters are information regarding a position or a posture of the reference cross section in the three-dimensional image. (Configuration 8) 8. The image processing apparatus according to claim 1, wherein the indices estimated by the indices estimation unit are indices that contribute to correction of the initial parameters within an image plane of the cross-sectional image. (Configuration 9) 9. The image processing device according to any one of configurations 1 to 8, wherein the index estimated by the index estimation unit is a correction amount for displacing a reference cross section obtained by the initial parameters. (Configuration 10) The image processing device described in configuration 9, characterized in that the correction amount is at least one of a displacement amount for moving a position of a reference cross section obtained by the initial parameters to a target position of the reference cross section, and a rotation amount for rotating a posture of the reference cross section obtained by the initial parameters to a target posture of the reference cross section. (Configuration 11) the indices estimated by the indices estimation unit are parameters for obtaining the reference cross section, The index estimation unit estimates parameters for obtaining the reference cross section based on the initial parameters. 9. The image processing device according to any one of configurations 1 to 8. (Configuration 12) 12. The image processing device according to claim 11, wherein the parameter for obtaining the reference cross section is at least one of an estimated value of a target position of the reference cross section and an estimated value of a target axis vector of the reference cross section. (Configuration 13) The image processing device according to configuration 11 or 12, wherein the landmark estimation unit estimates positions of a plurality of predetermined landmarks on the cross-sectional image, and estimates parameters for obtaining the reference cross-section based on the results of the position estimation. (Configuration 14) The image processing device according to any one of configurations 1 to 12, further comprising a correction unit that corrects the initial parameters based on the index estimated by the index estimation unit. (Configuration 15) The image processing device according to configuration 14, wherein the correction unit determines whether or not to correct the initial parameters based on the index estimated by the index estimation unit, and corrects the initial parameters based on a result of the determination. (Configuration 16) 16. The image processing device according to claim 15, wherein when there are a plurality of indices estimated by the index estimation unit, the correction unit determines for each index whether or not to correct the initial parameters based on the index. (Configuration 17) a display processing unit that performs processing to display the three-dimensional image, the cross-sectional image, and the indices estimated by the indices estimation unit, The correction unit acquires a user's selection result regarding which of the indices estimated by the index estimation unit is to be used as a basis for correcting the initial parameters, and corrects the initial parameters based on the selection result. 17. The image processing device according to any one of configurations 14 to 16, (Configuration 18) a display processing unit that performs processing to display the three-dimensional image, the cross-sectional image, and the indices estimated by the indices estimation unit, 17. The image processing device according to claim 16, wherein the correction unit corrects the indices estimated by the index estimation unit based on an instruction from a user, and corrects the initial parameters based on the corrected indices. (Configuration 19) the cross-sectional image is composed of a plurality of images whose positions and orientations have a predetermined relationship with each other; the cross-sectional image acquisition unit acquires a corrected cross-sectional image from the three-dimensional image based on the cross-sectional parameters corrected by the correction unit; The index estimation unit further estimates the index based on the corrected cross-sectional image. 15. The image processing device according to claim 14, (Configuration 20) The index estimation unit estimates the index based on a part of a plurality of images constituting the cross-sectional image, The index estimation unit sequentially switches, in a predetermined order, which of the plurality of images is to be used to estimate the index. 20. The image processing device according to claim 19, (Method 1) An image acquisition step of acquiring a three-dimensional image of the subject; an estimation step of estimating initial parameters, which are initial values ​​of parameters for obtaining a reference cross section, by reducing the resolution of the three-dimensional image; a cross-sectional image acquiring step of acquiring at least one cross-sectional image from the three-dimensional image based on the initial parameters; an index estimating step of estimating an index that contributes to correcting the initial parameters for each of the cross-sectional images; 13. An image processing method comprising: (Program 1) A program for causing a computer to execute the image processing method according to method 1. [Explanation of symbols]

[0111] 10 image processing device, 41 image acquisition unit, 42 initial parameter estimation unit, 43 cross-sectional image acquisition unit, 45 correction parameter estimation unit

Claims

1. an image acquisition unit for acquiring a three-dimensional image of a subject; an initial parameter estimation unit that estimates a plurality of initial parameters that are initial values ​​of a plurality of parameters for obtaining a plurality of reference cross sections from the three-dimensional image; a cross-sectional image acquisition unit that acquires at least one cross-sectional image from a plurality of cross-sectional images obtained from the three-dimensional image based on the plurality of initial parameters; an index estimation unit that estimates an index for correcting at least one of the plurality of initial parameters used to acquire a cross-sectional image different from the cross-sectional image among the plurality of cross-sectional images, from the cross-sectional image acquired by the cross-sectional image acquisition unit; a correction unit that corrects at least one of the plurality of initial parameters using the index estimated by the index estimation unit; An image processing device comprising:

2. The cross-sectional image acquisition unit acquires a first cross-sectional image and a second cross-sectional image from the three-dimensional image based on the plurality of initial parameters; The index estimation unit estimates, from the first cross-sectional image, a first index used to correct a first parameter used to acquire a cross-sectional image different from the first cross-sectional image, among the plurality of initial parameters, and estimates, from the second cross-sectional image, a second index used to correct a second parameter used to acquire a cross-sectional image different from the second cross-sectional image, among the plurality of initial parameters.

2. The image processing device according to claim 1, wherein:

3. The image processing device described in Claim 2, characterized in that the index estimation unit estimates the first index by inputting the first cross-sectional image into a first learning model, and estimates the second index by inputting the second cross-sectional image into a second learning model.

4. An image processing device described in any one of claims 1 to 3, characterized in that the cross-sectional image acquisition unit further acquires at least one cross-sectional image from the three-dimensional image using at least one of the initial parameters corrected by the correction unit.

5. 2. The image processing device according to claim 1, wherein the cross-sectional image acquisition unit acquires, from the three-dimensional image, a cross-sectional image including a plurality of slices corresponding to a predetermined thickness in the three-dimensional image, based on at least one of the plurality of initial parameters.

6. The initial parameter estimation unit estimates the plurality of initial parameters using the three-dimensional image with reduced resolution, The resolution of the cross-sectional image acquired by the cross-sectional image acquisition unit is higher than the resolution of the three-dimensional image whose resolution has been reduced.

2. The image processing device according to claim 1, wherein:

7. 4. The image processing apparatus according to claim 3, wherein the initial parameter estimation unit estimates the plurality of initial parameters based on the three-dimensional image with reduced resolution.

8. Further comprising a learning model acquisition unit for acquiring a learning model, the learning model is at least one of an initial parameter estimation learning model created by a learning process using information including a pair of a learning three-dimensional image and a parameter for obtaining the reference cross section in the learning three-dimensional image, and an index estimation learning model created by a learning process using information including a pair of a learning cross-sectional image and an index that contributes to correcting the reference cross-sectional parameters in the learning cross-sectional image, At least one of the initial parameter estimation unit and the index estimation unit performs the estimation based on the learning model.

2. The image processing device according to claim 1, wherein:

9. 2. The image processing apparatus according to claim 1, wherein the plurality of initial parameters are information about positions or orientations of the plurality of reference cross sections in the three-dimensional image.

10. The image processing device according to claim 9 , wherein the indices estimated by the indices estimation unit are indices used for correction within the image plane of the cross-sectional image acquired by the cross-sectional image acquisition unit.

11. 4. The image processing device according to claim 1, wherein the index estimated by the index estimation unit is a correction amount for displacing at least one of a plurality of reference cross sections obtained by the plurality of initial parameters.

12. 12. The image processing device according to claim 11, wherein the correction amount is at least one of a displacement amount for moving a position of at least one of the plurality of reference cross sections obtained by the plurality of initial parameters to a target position of the reference cross section, and a rotation amount for rotating an orientation of at least one of the reference cross sections obtained by the plurality of initial parameters to a target orientation of the reference cross section.

13. the index estimated by the index estimation unit is a parameter for obtaining at least one of the plurality of reference cross sections, The index estimation unit estimates a parameter for obtaining at least one of the plurality of reference cross sections based on at least one of the plurality of initial parameters.

4. The image processing device according to claim 1, wherein the image processing device is a computer.

14. 14. The image processing device according to claim 13, wherein the parameter for obtaining at least one of the plurality of reference cross sections is at least one of an estimated value of a target position of the at least one reference cross section and an estimated value of a target axis vector of the reference cross section.

15. 14. The image processing device according to claim 13, wherein the landmark estimation unit estimates positions of a plurality of predetermined landmarks on the cross-sectional image, and estimates parameters for obtaining at least one of the plurality of reference cross sections based on the results of the position estimation.

16. 2. The image processing device according to claim 1, wherein the correction unit determines whether or not to correct the initial parameters based on the indices estimated by the index estimation unit, and corrects the initial parameters based on a result of the determination.

17. 17. The image processing device according to claim 16, wherein when there are a plurality of indices estimated by the index estimation unit, the correction unit determines for each index whether or not to correct the initial parameters based on the index.

18. a display processing unit that performs processing to display the three-dimensional image, the cross-sectional image, and the indices estimated by the indices estimation unit, The correction unit acquires a user's selection result regarding which of the indices estimated by the index estimation unit is to be used to correct the initial parameters, and corrects the initial parameters based on the selection result.

18. The image processing device according to claim 17,

19. a display processing unit that performs processing to display the three-dimensional image, the cross-sectional image, and the indices estimated by the indices estimation unit, 18. The image processing device according to claim 17, wherein the correction unit corrects the indices estimated by the index estimation unit based on an instruction from a user, and corrects the initial parameters based on the corrected indices.

20. the cross-sectional image is composed of a plurality of images whose positions and orientations have a predetermined relationship between the images; the cross-sectional image acquisition unit acquires a corrected cross-sectional image from the three-dimensional image based on the initial parameters corrected by the correction unit; The index estimation unit further estimates the index based on the corrected cross-sectional image.

2. The image processing device according to claim 1, wherein:

21. the index estimation unit estimates the index based on a part of a plurality of images constituting the cross-sectional image; The index estimation unit sequentially switches, in a predetermined order, which of the plurality of images is used to estimate the index.

21. The image processing device according to claim 20.

22. an image acquisition step of acquiring a three-dimensional image of the subject; an estimation step of estimating a plurality of initial parameters, which are initial values ​​of a plurality of parameters for obtaining a plurality of reference cross sections from the three-dimensional image; a cross-sectional image acquiring step of acquiring at least one cross-sectional image from a plurality of cross-sectional images obtained from the three-dimensional image based on the plurality of initial parameters; an index estimating step of estimating an index for correcting at least one of the plurality of initial parameters used to acquire a cross-sectional image different from the cross-sectional image among the plurality of cross-sectional images, from the cross-sectional image acquired in the cross-sectional image acquiring step; a correction step of correcting at least one of the plurality of initial parameters using the index estimated in the index estimation step; An image processing method comprising:

23. A program for causing a computer to execute the image processing method according to claim 22.

Images