Image processing device and image processing method

Abstract

To provide a technique that can improve estimation performance in estimating reference cross sections for three-dimensional images.SOLUTION: An image processing device includes: an image acquisition unit for acquiring a three-dimensional image including a test subject as a photographic subject; an intersecting cross-section acquisition unit for acquiring information relating to a plurality of intersecting cross-sections having an intersecting relation with a prescribed reference cross-section, from the three-dimensional image; an intersecting line information acquisition unit for acquiring intersecting line information, the information relating to intersecting lines between the plurality of intersecting cross-sections and the reference cross-section, on the basis of the information relating to the plurality of intersecting cross-sections; and a cross-section information acquisition unit for acquiring reference cross-section information, the information relating to the reference cross-section, on the basis of the intersecting line information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image processing method.

Background Art

[0002] In diagnosis using medical images, a three-dimensional image (volume data) is cut based on a predetermined standard, a two-dimensional cross-section (reference cross-section) that is the cut surface is obtained, and an image of the cross-section (reference cross-section image) is displayed on a display or the like for diagnosis in some cases. Further, image processing may be performed on the reference cross-section image to measure the shape of an organ or detect an abnormality. However, manual setting of the reference cross-section involves the work of finding anatomical landmarks (feature points) that serve as clues from within three-dimensional space, which is a problem that places a heavy burden on doctors and the like.

[0003] To solve this problem, a technique has been proposed for calculating a highly accurate reference cross-section while reducing the burden of manual setting by correcting an initial value of the reference cross-section obtained by rough estimation or by a rough setting by a doctor or the like. In Patent Document 1, anatomical feature points are detected on peripheral cross-sections obtained starting from a certain reference cross-section, and the reference cross-section is corrected using an axis passing through the centers of those feature points. Further, in Patent Document 2, a user manually inputs an intersection line of a reference cross-section that intersects a certain cross-section, and the reference cross-section is updated using the intersection line information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the methods described in these prior arts, first, a predetermined starting point is determined, and a reference cross-section is calculated based on the peripheral information of the starting point. Therefore, for example, in an input three-dimensional image, when the image quality of the area around the starting point is low, it may not be possible to accurately estimate the reference cross-section.

[0007] The present disclosure has been made in view of the above, and an object thereof is to provide a technique capable of improving the estimation performance in the estimation of a reference cross-section for a three-dimensional image.

Means for Solving the Problems

[0008] The image processing apparatus according to the present disclosure includes an image acquisition unit that acquires a three-dimensional image including a subject as a photographed object, an intersection cross-section acquisition unit that acquires information related to a plurality of intersection cross-sections having an intersection relationship with a predetermined reference cross-section from the three-dimensional image, an intersection line information acquisition unit that acquires intersection line information, which is information related to an intersection line between the plurality of intersection cross-sections and the reference cross-section, based on the information related to the plurality of intersection cross-sections, and a cross-section information acquisition unit that acquires reference cross-section information, which is information related to the reference cross-section, based on the intersection line information.

[0009] In addition, the image processing method according to the present disclosure includes an image acquisition step of acquiring a three-dimensional image including a subject as a subject, an intersection cross-section acquisition step of acquiring information related to a plurality of intersection cross-sections having an intersection relationship with a predetermined reference cross-section from the three-dimensional image, an intersection line information acquisition step of acquiring intersection line information, which is information related to an intersection line between the plurality of intersection cross-sections and the reference cross-section, based on the information related to the plurality of intersection cross-sections, and a cross-section information acquisition step of acquiring reference cross-section information, which is information related to the reference cross-section, based on the intersection line information. The image processing method is characterized by including these steps.

Effects of the Invention

[0010] According to the technology of the present disclosure, in estimating a reference cross-section for a three-dimensional image, the estimation performance can be improved.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the present disclosure is not limited to the following embodiments and can be appropriately modified without departing from the gist thereof. In the drawings described below, components having the same function may be denoted by the same reference numerals, and the description thereof may be omitted or simplified.

[0013] The image processing apparatus according to the embodiment described below provides a function of estimating a predetermined reference cross-section for a doctor or the like to observe (diagnose) an input three-dimensional image. The input image to be processed is a medical image, that is, an image including a subject (such as a human body) photographed or generated for purposes such as medical diagnosis, examination, and research, and is typically an image acquired by an imaging system called a modality. Examples of the input image include ultrasonic images obtained by an ultrasonic diagnostic apparatus. The input image may also be an X-ray CT (Computed Tomography) image obtained by an X-ray CT apparatus, an MRI (Magnetic Resonance Imaging) image obtained by an MRI apparatus, and the like.

[0014] In the following description, a case where a trans-sternal three-dimensional ultrasonic image obtained by imaging the right ventricular region of the heart is used as an input image and a reference cross-section of the right ventricular region to be observed is estimated will be described as an example.

[0015] <First Embodiment> The image processing apparatus according to the first embodiment uses a three-dimensional image as an input image and determines parameters (reference cross-section parameters) representing the position and orientation of a reference cross-section for observing or analyzing the right ventricle. It is estimated. At this time, first, the reference cross-section parameters are estimated (i.e., "roughly estimated") based on the image with the reduced resolution of the input 3D image, and the roughly estimated reference cross-section (the reference cross-section before update) is calculated. Next, a plurality of cross-sections (cross-section group) that intersect the reference cross-section before update are obtained. Then, at each cross-section of the cross-section group, an "intersection line" representing the position where it intersects the reference cross-section after update (i.e., the final reference cross-section) is estimated. Finally, using the information of each intersection line estimated in this way, the parameters of the reference cross-section after update are calculated.

[0016] Figure 3A shows the definition of the reference cross-section in this embodiment. The reference cross-section 310 is a cross-section in the space of the input 3D image 301 defined based on the anatomical structure of the subject. Also, the reference cross-section image 320 shown in Figure 3B is a 2D cross-section image obtained by cutting out the reference cross-section from the input 3D image 301. Hereinafter, the parameters of the reference cross-section may sometimes be simply referred to as the "reference cross-section". As shown in Figure 3B, the reference cross-section in this embodiment is an apical four-chamber view that can simultaneously observe the four chambers of the left ventricle 311, left atrium 312, right ventricle 313, and right atrium 314. In addition to this, instead, the reference cross-section may be a short-axis view of the right ventricle related to the right ventricle 313. Also, its central position 315 is the midpoint of two points 316 and 317 (hereinafter referred to as the "left and right annulus midpoint") representing the annulus position drawn on the cross-section when the annulus of the tricuspid valve is cut by the reference cross-section.

[0017] Furthermore, it is assumed that the vertical direction (indicated by the dashed line 318) of the reference cross-section coincides with the direction connecting the probe position and the maximum depth position of the ultrasonic signal. Here, the central position of the reference cross-section is represented by the 3D coordinate values (cx, cy, cz) in the image coordinate system of the input 3D image. Also, the orientation of the reference cross-section is represented by the rotation angles (α, β, γ) around each coordinate axis in the image coordinate system of the input 3D image. That is, the position and orientation of the reference cross-section are represented by a total of six parameters.

[0018] In addition, the representation of the pose by the rotation angles around each coordinate axis in the image coordinate system of the input three-dimensional image shown above is mutually convertible with the representation of the pose by three mutually orthogonal unit vectors (normal vector, cross-section left-right direction vector, cross-section up-down direction vector). For convenience of explanation, hereinafter, the notation using the normal vector (nx, ny, nz), the left-right direction vector (sx, sy, sz), and the up-down direction vector (lx, ly, lz) may be used. Note that the pose may be obtained by a value expressed by any method of representing the pose other than the rotation angles around the above-mentioned coordinate axes. For example, a quaternion may be used, or it may be represented by a combination of a rotation axis vector and a rotation angle around the axis. Also, the reference cross-section (reference cross-section before update) calculated prior to the cross-section group and the reference cross-section (reference cross-section after update) finally calculated based on the cross-section are different cross-sections.

[0019] Figures 4A to 4C schematically show the cross-section group and the cross-section image group in the present embodiment. The cross-section group is a plurality of cross-sections that intersect with the reference cross-section in a predetermined relationship. The cross-section group of this example is a plurality of cross-sections that are orthogonal to the reference cross-section before update, parallel to the left-right direction (X-axis direction) of the reference cross-section before update, and cut the right ventricle and the left ventricle one by one. In this example, two cross-sections 402 and 403 intersect the reference cross-section image 320 in the positional relationship shown in Figure 4A. The cross-section images 420 and 430 are two-dimensional cross-section images cut out from the input three-dimensional image 301 for the respective cross-sections 402 and 403. Examples of the two cross-section images are illustrated in Figures 4B and 4C. Similar to the case of the reference cross-section, each cross-section image is also a two-dimensional image calculated from the parameters representing the cross-section (simply referred to as the cross-section) and the input three-dimensional image. The cross-section image 420 in Figure 4B is an image of the cross-section 402 passing through the midpoint 415 of the left and right valve rings in Figure 4A. Similarly, the cross-section image 430 in Figure 4C is an image of the cross-section 403 passing through the midpoint 416 between the midpoint 415 of the left and right valve rings and the upper end of the image in Figure 4A. Similar to the reference cross-section, the position and orientation of each cross-section are also represented by six parameters.

[0020] Next, Figures 5A to 5D are used to calculate the reference cross-section after update using a plurality of cross-sections. This will explain the process. FIGS. 5A and 5B show cross-sections, which are the same cross-sections as FIGS. 4B and 4C respectively. Also, FIGS. 5C and 5D show the reference cross-section 510 before update and the reference cross-section 520 after update within the input 3D image 501 space. The image processing apparatus according to the present embodiment uses the images of these plurality of cross-sections as inputs to estimate the positions of the points (points 502, 503 in FIG. 5A and points 504, 505 in FIG. 5B) where the intersection lines with the reference cross-section intersect the right ventricular contour. Then, as shown in FIG. 5D, a plane that best fits the thus-estimated point group is calculated. The reference cross-section before update (cross-section 510 in FIG. 5C) is updated with the plane calculated in this way, and the updated reference cross-section (cross-section 520 in FIG. 5D) is obtained. The specific process will be described in detail in the explanation of step S205.

[0021] Hereinafter, the configuration and processing of the image processing apparatus according to the present embodiment will be described. FIG. 1 is a block diagram showing a configuration example of an image processing system (also referred to as a medical image processing system) including the image processing apparatus according to the present embodiment. The image processing system 1 includes an image processing apparatus 10 and a database 22. The image processing apparatus 10 is connected to the database 22 via a network 21 in a communicable state. The network 21 includes, for example, a LAN (Local Area Network) or a WAN (Wide Area Network).

[0022] The database 22 holds and manages a plurality of images and information used in the processes described below. The information managed by the database 22 includes information on the input 3D images that are the targets of the cross-section parameter estimation process in the image processing apparatus 10. The image processing apparatus 10 can acquire the data held in the database 22 via the network 21. When the cross-section parameter estimation process and the inner membrane contour estimation process in the image processing apparatus 10 are performed based on an inference model, the information of the inference model is managed by the database 22. Note that the information of the inference model may be stored in the internal memory (ROM 32 or storage unit 34) of the image processing apparatus 10 instead of the database 22.

[0023] The image processing apparatus 10 includes a communication IF (Interface) 31, a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33, a storage unit 34, an operation unit 35, a display unit 36, and a control unit 37.

[0024] The communication IF 31 is constituted by a LAN card or the like, and is a communication unit that realizes communication between the image processing apparatus 10 and an external device (for example, the database 22 or the like). The ROM 32 is constituted by a non-volatile memory or the like, and stores various programs and various data. The RAM 33 is constituted by a volatile memory or the like, and is used as a work memory that temporarily stores a program and data being executed. The storage unit 34 is constituted by an HDD (Hard Disk Drive) or the like, and stores various programs and various data. The operation unit 35 is constituted by a keyboard, a mouse, a touch panel, or the like, and inputs instructions from a user (for example, a doctor or a medical technician) to various devices. The display unit 36 is constituted by a display or the like, and displays various information to the user.

[0025] The control unit 37 is constituted by a CPU (Central Processing Unit) or the like, and comprehensively controls the processing in the image processing apparatus 10. As its functional configuration, the control unit 37 includes an image acquisition unit 41, a cross-section parameter estimation unit 42, a cross-section group acquisition unit 43, a cross-line estimation unit 44, a cross-section information update unit 45, and a display processing unit 51. The control unit 37 may include a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), or the like.

[0026] The image acquisition unit 41 is a three-dimensional input image of a subject input to the image processing apparatus 10, the input three-dimensional The original image is acquired from the database 22. The details of the process will be described in detail in the description of step S201. Note that the input 3D image may be directly acquired from the modality. In that case, the image processing apparatus 10 may be implemented within the console of the modality (imaging system).

[0027] The cross-sectional parameter estimation unit 42 estimates parameters for obtaining a reference cross-section from the input 3D image acquired by the image acquisition unit 41. What is obtained here is a rough estimation result of the reference cross-section (parameters of the reference cross-section before update). The details of the process will be described in detail in the description of step S202.

[0028] The cross-sectional image group acquisition unit 43 acquires a two-dimensional cross-sectional image group (cross-sectional image group) representing a plurality of cross-sections (cross-sectional image group) in an intersecting relationship with the reference cross-section based on the input 3D image acquired by the image acquisition unit 41 and the parameters of the reference cross-section before update estimated by the cross-sectional parameter estimation unit 42. The details of the process will be described in detail in the description of step S203.

[0029] The intersection line estimation unit 44 is a cross-sectional image acquisition unit that acquires information related to a cross-section in an intersecting relationship with a predetermined reference cross-section from a 3D image. The intersection line estimation unit 44 estimates information (intersection line information) regarding the intersection line of each cross-section of the reference cross-section using each of the cross-sectional image groups acquired by the cross-sectional image group acquisition unit 43. The details of the process will be described in detail in the description of step S204.

[0030] The cross-sectional information update unit 45 is a cross-sectional information acquisition unit that acquires reference cross-sectional information, which is information related to the reference cross-section, based on the intersection line information. The cross-sectional information update unit 45 calculates the updated reference cross-section parameters using at least the intersection line information estimated by the intersection line estimation unit 44. The details of the process will be described in detail in the description of step S205.

[0031] The display processing unit 51 displays the information processed by the image processing apparatus 10, such as the input 3D image and the reference cross-section parameters, in a display form that can be easily visually recognized by the user of the image processing apparatus 10 in the image display area of the display unit 36. The details of the process will be described in detail in the description of step S206.

[0032] Each component of the above-described image processing apparatus 10 functions according to a computer program. For example, the control unit 37 (CPU) reads and executes a computer program stored in the ROM 32 or the storage unit 34 with the RAM 33 as a work area, thereby realizing the functions of each component. Note that some or all of the functions of the components of the image processing apparatus 10 may be realized by using dedicated circuits. Also, some of the functions of the components of the control unit 37 may be realized by using cloud computing technology. For example, an arithmetic unit located at a location different from the image processing apparatus 10 may be communicably connected to the image processing apparatus 10 via the network 21. Then, by the image processing apparatus 10 transmitting and receiving data to and from the arithmetic unit, the functions of the components of the image processing apparatus 10 or the control unit 37 may be realized.

[0033] Next, an example of the process executed by the image processing apparatus 10 in FIG. 1 will be described using the flowchart of FIG. 2.

[0034] (Step S201: Acquisition of input image) In step S201, the image processing apparatus 10 acquires an instruction for image acquisition from the user via the operation unit 35. Then, the image acquisition unit 41 acquires the input 3D image specified by the user from the database 22 and stores it in the RAM 33. Note that, in addition to acquiring the input 3D image from the database 22, an input image may be acquired from among ultrasonic images continuously captured by the ultrasonic diagnostic apparatus.

[0035] (Step S202: Estimation of reference cross-section parameters) In step S202, the cross-section parameter estimation unit 42 estimates parameters that define the center position and orientation of the reference cross-section with the input 3D image as an input. The cross-section parameter estimation unit 42 is an estimation result acquisition unit that acquires an estimation result of information related to the reference cross-section from the 3D image. As described above, the reference cross-section parameters are a set of parameters (three parameters each, for a total of six parameters) that represent the position and orientation of the reference cross-section in the image coordinate system of the input 3D image.

[0036] In this embodiment, the estimation of the reference cross-sectional parameters uses a method based on a convolutional neural network (CNN). That is, in advance, the relationship between the reference cross-sectional parameters for three-dimensional ultrasonic images obtained by imaging the right ventricular region is learned using a CNN. Then, in the processing of this step, the learned CNN is used to estimate the reference cross-sectional parameters from the input three-dimensional image.

[0037] Here, the three-dimensional image input to the CNN is not the same as the input three-dimensional image obtained in step S201, but a "coarse" image with reduced resolution of the input three-dimensional image. For example, assume that the input three-dimensional image is a volume image representing a range of 0.6 mm per voxel and 256×256×256 voxels, that is, each side is 153.6 mm. In this step, the length per voxel is made four times that of the original image, and this input three-dimensional image is represented by 64×64×64 voxels. That is, the represented range remains 153.6 mm per side, but the length per voxel is converted to an image of 2.4 mm. By doing so, the calculation time and memory usage required for inference using the CNN can be reduced compared to the case of using the input three-dimensional image as it is. Here, the three-dimensional image input to the CNN may be further subjected to known image processing such as pixel value normalization and contrast correction using the mean and variance of the pixel values as preprocessing. Note that the above-described resolution conversion process may use any known method. For example, the voxel values can be sampled at intervals according to the degree of resolution reduction, or the pixel values of voxels in a range according to the degree of resolution reduction can be averaged. Also, the image processing such as the above-described resolution conversion process and pixel value normalization can be executed in any order, and these processes do not necessarily have to be performed.

[0038] In this embodiment, the position and orientation of the reference cross-section are represented by six parameters. In the processing of this step, the case where the above six values are directly estimated as the output of the estimation by the CNN is described as an example. However, in this embodiment, as the output of the estimation by the CNN, any expression form representing the position and orientation may be used, and the cross-section parameter estimation unit 42 may be a position and orientation acquisition unit that acquires information including at least either the position or the orientation of the subject from the three-dimensional image.

[0039] Further, the processing of this step may not be the estimation by the image processing apparatus 10 based on the input three-dimensional image, but the control unit 37 may acquire the setting content input by the user via the operation unit 35 and use the acquired setting content as the reference cross-section parameter before update.

[0040] (Step S203: Calculation of cross-section image group) In step S203, the cross-section group acquisition unit 43 calculates the parameters of a plurality of cross-sections (cross-section group) that intersect with the reference cross-section before update, using the input three-dimensional image and the reference cross-section parameters before update estimated in step S202. Then, based on the parameters of these cross-section groups, a cross-section image group in which the regions of each cross-section are cut out from the input three-dimensional image is calculated. Here, it is assumed that the position and orientation of each cross-section are defined by the relative positional relationship with respect to the reference cross-section before update. That is, the parameters of each cross-section can be uniquely calculated from the parameters of the reference cross-section before update.

[0041] The method of calculating the parameters of the cross-section group will be described with reference to FIG. 4. In the description of FIG. 4 , as the set of cross-sections, use two cross-sections: the first cross-section (the first cross-section) and the second cross-section (the second cross-section). Also, in this embodiment, each cross-section is parallel to each other (i.e., the postures are common), and only the central positions are different. First, calculate the parameters of the posture that are common to all cross-sections. This is calculated based on the posture parameters of the reference cross-section before update calculated in step S202. Specifically, the normal direction of the cross-section is the up-down direction of the reference cross-section before update, and the up-down direction of the cross-section is the normal direction of the reference cross-section before update. And the left-right direction of the cross-section is the same as the left-right direction of the reference cross-section before update. That is, each cross-section is orthogonal to the reference cross-section before update.

[0042] Next, determine the central position of each cross-section. Among the set of cross-sections composed of a plurality of cross-sections, the central position of the first cross-section is the same as the central position of the reference cross-section before update. That is, it is the same as the midpoint 415 of the left and right valve rings. The central position of the second cross-section is the position obtained by translating the central position of the first cross-section parallel along the normal direction of the cross-section. The central position of the second cross-section is the midpoint 416 between the central position of the first cross-section and the position extended from this central position to the upper end of the reference cross-section before update in the normal direction.

[0043] After calculating the parameters of all cross-sections in this way, cut out the two-dimensional cross-sectional images of each cross-section from the input three-dimensional image. The two-dimensional cross-sectional image is calculated by sampling the input three-dimensional image using the central position, left-right direction vector, and up-down direction vector calculated in this step. Here, the length per pixel is the same as that of the input three-dimensional image (for example, 0.6 mm), and sample the range of 256×256 pixels (153.6 mm×153.6 mm when the length per pixel is 0.6 mm). Then, store the obtained two-dimensional cross-sectional image in the RAM 33. Here, when the sampling range reaches outside the defined range of the input three-dimensional image, insert the pixel value "0" at that position.

[0044] In addition, in this embodiment, the case of using two cross-sections has been described as an example. However, for example, three or more cross-sections obtained by equally dividing the interval between the first cross-section and the second cross-section by a predetermined number may be used. In this case, the third and subsequent cross-sections are planes parallel to the first cross-section and the second cross-section.

[0045] (Step S204: Estimation of intersection line information) In step S204, the intersection line estimation unit 44 estimates information representing the intersection line with the reference cross-section in each cross-section image using the group of cross-section images acquired in step S203. The intersection line in this embodiment is a line that passes through the vicinity of the centers of both the right ventricle 511 and the left ventricle 513. As shown in FIGS. 5A and 5B, in this embodiment, the information representing the intersection line is represented by the intersection points (that is, two points for each cross-section image) when the intersection line intersects the contour of the right ventricle region. In step S204, the intersection line estimation unit 44 is an intersection line information acquisition unit that acquires intersection line information, which is information related to the intersection lines between a plurality of cross-sections and the reference cross-section, based on information related to the plurality of cross-sections.

[0046] In this embodiment, the estimation of the intersection points is performed using a CNN. That is, in advance, a large number of sets of 3D images and information on the reference cross-section set by a doctor or the like are collected, and the cross-section images and the positions of the above intersection points in the cross-sections are calculated. Then, a CNN that learns the relationship between the cross-section images and the positions of the above intersection points is constructed using those information. And in the process of this step, the coordinates of the intersection points are estimated from the cross-section images using the learned CNN. The CNN performs learning and estimation for each of the plurality of cross-section images. That is, when there are two cross-section images, CNNs for performing inference for each of the cross-section images are constructed respectively, and inferences of the intersection points are performed from each cross-section image using those CNNs.

[0047] In this way, by the inference process based on the 3D image, information (points 502, 503 in FIG. 5A and points 504, 505 in FIG. 5B) representing the intersection line of the reference cross-section in each cross-section image is estimated.

[0048] In this embodiment, the case where the CNN is learned and used for inference for each cross-section has been described as an example. However, the learning and inference for all cross-sectional images may be executed by a single CNN. In that case, it becomes possible to save the data storage capacity occupied by the CNN.

[0049] (Step S205: Update of reference cross-section parameters) In step S205, the cross-section information update unit 45 updates the reference cross-section parameters based on the intersection line information estimated in step S204.

[0050] The update process of the cross-section information will be described with reference to FIGS. 5A to 5D. Points 502, 503, 504, and 505 in FIGS. 5A and 5B are intersection points estimated two by two for each cross-section in step S204. In this step, an approximate plane passing through these points is calculated using the least squares method, which is a known technique. Thereby, the attitude component of the updated reference cross-section is determined. The vertical direction in the updated reference cross-section is the same as that before the update and coincides with the direction connecting the probe position and the maximum depth position of the ultrasonic signal. Here, the two points 502 and 503 estimated on the first cross-section (FIG. 5A) are points on the valve annulus. Therefore, based on the definition of the center position of the cross-section shown in FIG. 3B, the center position in the updated reference cross-section is the midpoint between point 502 and point 503. By doing so, as shown in the plane 520 of FIG. 5D, the updated reference cross-section is calculated. Further, an approximate plane may be obtained under the condition that the left and right valve annulus center points in the reference cross-section before the update are included (that is, the center position is not changed from the reference cross-section before the update). In this embodiment, the case where there are two cross-sections has been described as an example. However, even when there are three or more cross-sections, the reference cross-section parameters can be updated by the same method.

[0051] Note that it is also possible to update the reference cross-section parameters based on both the reference cross-section parameters before update estimated in step S202 and the intersection line information estimated in step S204. In this case, the center position of the reference cross-section parameters before update is maintained, and only the normal vector is updated. In this case, first, the "normal vector in each intersection cross-section" is calculated from the intersection line information estimated in step S204. Specifically, this normal vector is the vector of the axis in the intersection cross-section perpendicular to the straight line connecting point 502 and point 503 in FIG. 5A, and is the vector of the axis in the intersection cross-section perpendicular to the straight line connecting point 504 and point 505 in FIG. 5B.

[0052] Next, an average normal vector is calculated from these normal vectors. When the normal vectors calculated for each of the plurality of intersection cross-sections are close to each other, the normal vectors may be simply averaged. Alternatively, those with an angular difference from the normal vector in the reference cross-section before update of a certain degree or more (for example, 10 degrees or more) may be excluded as outliers and then averaged. The vector calculated in this way is replaced with the normal vector of the reference cross-section before update to update the reference cross-section parameters. And for the vertical direction in the plane, it is the same as the reference cross-section before update. By doing so, since there is no change from the reference cross-section before update regarding the center position, it is possible to prevent a significant deviation from the reference cross-section before update.

[0053] (Step S206: Display of processing result) In step S206, the display processing unit 51 displays the processing result information of the image processing apparatus 10 in a display form that can be easily visually recognized by the user within the image display area of the display unit 36. The information to be displayed shall include at least the reference cross-section image cut out by the reference cross-section parameters updated in step S205. At this time, the information to be displayed may also include information calculated from the reference cross-section, such as an orthogonal cross-section image orthogonal to the reference cross-section. It is okay.

[0054] In addition, when the purpose is analysis or measurement based on a reference cross-section, the display process in step S206 is not essential. It may be configured to save the cross-section parameters obtained in step S205 in a storage device or output them externally. Further, when an analysis unit (not shown) that performs analysis or measurement using the reference cross-section is provided in the image processing apparatus 10, it may be configured to transmit the input 3D image and the information of the reference cross-section parameters to the analysis unit.

[0055] According to the present embodiment, when estimating parameters (reference cross-section parameters) representing the position and orientation of a reference cross-section for observing the right ventricle using a 3D image as an input image, the reference cross-section parameters are updated using the information of the cross-sections that intersect the roughly estimated reference cross-section. Thereby, it becomes possible to improve the accuracy of estimating the reference cross-section.

[0056] In the first embodiment, when calculating a plurality of intersecting cross-section images in step S203, each intersecting cross-section is orthogonal to the reference cross-section and is parallel to each other. However, as long as each intersecting cross-section has a predetermined positional relationship with the reference cross-section before update and each intersecting cross-section does not intersect at the same position as the reference cross-section before update, this condition does not necessarily have to be satisfied. For example, the intersecting cross-section may intersect the reference cross-section at an angle other than 90 degrees (for example, 80 degrees), or the intersecting cross-sections may not be parallel to each other. Also, the positional relationship with the reference cross-section does not have to be a predetermined value, and may be adaptively set according to the 3D image to be processed. For example, the normal vectors of each intersecting cross-section may deviate from the parallel relationship within a range of a predetermined value (for example, ±5 degrees). For example, the contrast (the difference between the minimum pixel value and the maximum pixel value) of the calculated intersecting cross-section image may be measured, and an intersecting cross-section image with the maximum contrast may be obtained under the condition that the intersecting cross-sections do not intersect each other and the rotation amount of the normal vector is within a predetermined value (for example, ±5 degrees). In this case, since a cross-section image with high contrast and easy to identify the tissue pattern is input to the estimation process in step S204, it becomes possible to improve the estimation accuracy of the intersection line estimation.

[0057] Also, regarding the intersection position with the reference cross-section, instead of using a fixed value, for example, a plurality of candidate values within a position range based on a predetermined positional relationship between the reference cross-section before update and a predetermined position may be used to obtain the cross-sectional image. Then, for example, the contrast of the cross-sectional image may be measured in the same manner as described above, and the cross-sectional image with the maximum contrast within the position range may be obtained. Also in this case, since a cross-sectional image with high contrast and easy to identify the tissue pattern is input to the estimation process in step S204, it is possible to improve the estimation accuracy of the intersection line estimation.

[0058] In the first embodiment, the reference cross-sections before and after the update were represented by six parameters (three position parameters and three attitude parameters). However, only one of the position and the attitude may be used as the parameter of the reference cross-section. In this case, the cross-sectional plane intersecting the reference cross-section is calculated by combining the reference cross-section parameter and the coordinate system of the input three-dimensional image. For example, when the reference cross-section parameter is only the position, the normal vector of the first cross-section in step S203 is the vertical direction vector of the input three-dimensional image, and the center position is the center position of the reference cross-section before the update. On the other hand, when the reference cross-section parameter is only the attitude, the first cross-section is a cross-section orthogonal to the reference cross-section parameter before the update and passing through the center point of the input three-dimensional image. In either case, the second cross-section is a cross-section obtained by translating the first cross-section parallel in the normal direction by a predetermined amount (for example, 10 mm). The process of estimating the intersection line is the same as the process of step S204 in the first embodiment, and the update amount of either the position or the attitude is calculated from the intersection line information. By doing so, since the position and the attitude of the reference cross-section can be estimated independently, for example, when the user manually inputs information on either the position or the attitude, it is possible to flexibly respond.

[0059] In the first embodiment, as the intersection line information estimated in step S204, intersections where the intersection line intersects the right ventricular contour (two points per intersection cross-section) were estimated. However, the intersection line information estimated in step S204 is not limited to this. For example, a two-dimensional vector representing the "orientation" of the intersection line on the intersection cross-section, information representing a "line", that is, a set of coordinate values representing a two-dimensional vector and a position on the line may be estimated. In that case, the cross-section information update method in step S205 is also changed. For example, the updated reference cross-section parameters are calculated by iterative optimization. In this case, the reference cross-section parameters (six parameters) are used as variables to be adjusted for optimization. And in each step of iterative optimization, first, an intersection line based on the geometric relationship between the cross-section represented by the reference cross-section parameters and each intersection cross-section is calculated. These are represented as two-dimensional vectors on each intersection cross-section. Then, the angle difference between each two-dimensional vector and the intersection line vector estimated in step S204 is calculated. The cost in iterative optimization is the total value of the angle differences calculated in this way. By doing so, since it is not necessary to pinpoint the intersection of the right ventricular region and the intersection line, it is possible to reduce the risk of failure in estimating the end points due to noise in the intersection cross-section image or the like.

[0060] In the first embodiment, when calculating the intersection cross-section image in step S203, the left-right direction of the intersection cross-section was made to coincide with the left-right direction of the reference cross-section before update. However, the left-right direction of the intersection cross-section may be determined without relying on the reference cross-section before update. In this case, the left-right direction of the intersection cross-section is set to coincide with the left-right direction (X-axis direction) of the input three-dimensional image. As a result, the intersection cross-section is not affected by the estimation accuracy of the in-plane components in the reference cross-section before update, and thus can operate stably.

[0061] <Modification Example 1 of the First Embodiment> Hereinafter, the modification examples of the first embodiment described above will be explained. In the first embodiment, an example where the processing target is a three-dimensional ultrasonic image of the right ventricular region of the heart was shown. However, the technology of the present disclosure can also be implemented when using images of regions of the heart other than the right ventricle, images of other organs other than the heart, or images obtained by other modalities.

[0062] As an example of applying the present invention to an image obtained by photographing a region other than the right ventricular region of the heart with a modality other than an ultrasonic device, there is a case of generating a developed image obtained by cutting open a tubular region such as the aorta from a three-dimensional X-ray CT image. In this case, the cutting plane for cutting open the tubular region is defined as the reference cross-section, and the cutting plane for making a circular cut of the tubular region is defined as the intersecting cross-section. Here, the center position of each intersecting cross-section can be set to the centroid position of the contour of the tubular region. And the estimated intersection line can be set such that the centroid position is one end point and the position where the cutting plane for cutting open the tubular region intersects the contour of the tubular region is the other end point.

[0063] Thus, according to this modification example, the technology of the present disclosure can be applied to images of modalities other than three-dimensional ultrasonic images and to objects other than the right ventricular region of the heart.

[0064] <Modification Example 2 of the First Embodiment> Next, another modification example of the first embodiment will be described. In the first embodiment, when performing estimation in step S202 and step S204, a CNN is used as the estimation algorithm, but it is also feasible to use another estimation algorithm.

[0065] As an estimation algorithm other than CNN, for example, a method based on PCA adopted in the above-mentioned prior art or a deep learning method other than CNN such as a vision transformer may be used. Also, a machine learning method not based on deep learning such as regression using a support vector machine may be used. Also, different estimation algorithms may be used in the processing of step S202 and step S204. Thus, according to this modification example, it is possible to execute the processing using various methods, not limited to CNN.

[0066] <Second Embodiment> Next, the second embodiment will be described. Similar to the first embodiment, the image processing apparatus according to the second embodiment estimates parameters (reference cross-section parameters) representing the position and orientation of a reference cross-section for observing the right ventricle, using a three-dimensional image as the input image. In the first embodiment, all of the plurality of cross-sections defined for correcting the reference cross-section are parallel to the left-right direction (X-axis direction) of the reference cross-section before update. That is, it does not necessarily follow the variations in the posture of the right ventricle with respect to the reference cross-section before update. In this embodiment, a group of cross-sections is set using the vector of the "central axis" estimated based on the posture of the right ventricle on the reference cross-section before update. By doing so, it becomes possible to calculate the group of cross-sections based on the anatomical structure of the right ventricle.

[0067] Figures 7A to 7C schematically show the reference cross-section and the central axis in this embodiment, and a plurality of cross-sections calculated based on the central axis. In Figure 7A, points 714 and 716 are the positions where the tricuspid valve annulus intersects the reference cross-section 701, and point 715 is the midpoint of these two points (midpoint of the left and right valve annuli). In this embodiment, the central axis vector 710 is a vector of an axis on the reference cross-section 701, passes through the midpoint of the left and right valve annuli 715, and is a vector of an axis orthogonal to the axis connecting points 714 and 716. The central axis set in this way is also called the "right ventricular inflow tract" and is an axis that defines the posture of the right ventricle. The up-down direction of the reference cross-section in this embodiment is assumed to coincide with this central axis.

[0068] The planes 702 and 703 shown by the dashed lines in Figure 7A are an example of the cross-sections in this embodiment, and are cross-sections orthogonal to both the reference cross-section 701 and the central axis vector 710. By setting the cross-sections based on this axis, the cross-section images (Figures 7B and 7C) become sliced images reflecting the posture of the right ventricle in the input three-dimensional image, and the variation between cases of the images drawn on the cross-sections becomes smaller. Therefore, it becomes easier to perform the inference process of calculating the intersection lines from this cross-section, and it becomes possible to perform more appropriate correction of the reference cross-section parameters.

[0069] The configuration and processing of the image processing apparatus according to the present embodiment will be described. The configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the first embodiment (FIG. 1). Further, the processing executed by the image acquisition unit 41, the intersection line estimation unit 44, and the display processing unit 51 is the same as that of the first embodiment. In the description of the present embodiment, the same components and processes as those in the first embodiment are denoted by the same reference numerals, and detailed descriptions thereof are omitted.

[0070] The cross-section parameter estimation unit 42 estimates, in the same manner as in the first embodiment, parameters for obtaining a reference cross-section before update from the input three-dimensional image acquired by the image acquisition unit 41. Further, in the present embodiment, on the reference cross-section before update, a central axis vector that determines the posture of the right ventricle is estimated. Details of the processing will be described in detail in the descriptions of step S602 and step S603.

[0071] The cross-section group acquisition unit 43 acquires a two-dimensional cross-section image representing a plurality of cross-sections (intersecting cross-sections) in an intersecting relationship with the reference cross-section based on the input three-dimensional image acquired by the image acquisition unit 41, the parameters of the reference cross-section before update estimated by the cross-section parameter estimation unit 42, and the central axis vector. Details of the processing will be described in detail in the description of step S604.

[0072] The cross-section information update unit 45 calculates updated reference cross-section parameters using the initial reference cross-section parameters and the central axis vector estimated by the cross-section parameter estimation unit 42, and the cross-line information estimated by the cross-line estimation unit 44. Details of the processing will be described in detail in the description of step S606.

[0073] Next, an example of the processing of the image processing apparatus 10 in FIG. 1 in the present embodiment will be described using the flowchart of FIG. 6. Here, the processing of steps S601, S602, S605, and S607 is the same as the processing of steps S201, S202, S204, and S206 in the flowchart of the first embodiment (FIG. 2), respectively. Hereinafter, only the processing steps different from those in the first embodiment will be described.

[0074] (Step S603: Estimation of Feature Information) In step S603, the cross-section parameter estimation unit 42 estimates the feature information necessary for calculating the cross-sectional image group in step S604 based on the reference cross-section before update estimated in step S602. In the present embodiment, the feature information is the central axis vector that determines the posture of the right ventricle. In step S603, the cross-section parameter estimation unit 42 is a central axis acquisition unit that acquires information related to the central axis representing the orientation of the subject in the three-dimensional image. As shown in FIG. 7A, the central axis vector in the present embodiment is defined as an axis passing through the midpoint 715 of the left and right valve rings and orthogonal to the axis connecting the tricuspid valve ring points 714 and 716.

[0075] A method for estimating the central axis vector in this step is specifically shown. First, a cross-sectional image of the reference cross-section before update is calculated based on the parameters of the reference cross-section before update (a total of six parameters of the central position and posture) estimated in step S602. This is the same as the process of cutting out a two-dimensional cross-sectional image from the input three-dimensional image in step S203 of the first embodiment. Next, using the two-dimensional cross-sectional image thus obtained as input, the coordinate positions of the tricuspid valve ring points (two points) are estimated. The estimation is performed using a CNN. Finally, using the two estimated tricuspid valve ring points, the central axis vector is calculated based on the above definition.

[0076] Note that when estimating the coordinate positions of the two tricuspid valve ring points, not only the coordinates of the two points but also the contour of the right ventricle may be estimated, and based on that, the coordinate positions of the tricuspid valve ring points may be estimated. Specifically, the positions of a large number of point groups forming the contour of the right ventricle are estimated simultaneously, and the desired two-point coordinates are calculated by taking out the two end points of these point groups. The estimation is based on, for example, a CNN, and a model pre-learned for performing the estimation is used. When estimating the two tricuspid valve ring points based on the overall contour of the right ventricle, stable calculation can be performed because it is based on a continuous shape. Specifically, when using a CNN, in the design of the loss function during learning, the relative positional relationship of neighboring points can be used, so that more accurate estimation can be performed.

[0077] (Step S604: Calculation of cross-sectional image group) In step S604, the intersection cross-section group acquisition unit 43 acquires a group of intersection cross-section images obtained by cutting the input 3D image in a plurality of cross-sections (intersection cross-section group) that are in an intersecting relationship with the reference cross-section. The group of intersection cross-section images is calculated based on the input 3D image acquired by the image acquisition unit 41, the parameters of the reference cross-section before update estimated by the cross-section parameter estimation unit 42, and the central axis vector. Each of the group of intersection cross-section images is a 2D image.

[0078] In step S203 of the first embodiment, the normal vectors of all the intersection cross-sections are the same as the vertical direction of the reference cross-section before update. On the other hand, in this embodiment, the normal vector of the intersection cross-section is made to coincide with the central axis vector estimated in step S603. Also, the central position of each intersection cross-section is assumed to be the position where the central axis vector passes through each intersection cross-section. Except for the above points, the processing content of this step is common to the processing content of step S203 of the first embodiment.

[0079] (Step S606: Update of reference cross-section parameters) In step S606, the cross-section information update unit 45 calculates the updated reference cross-section parameters using the initial reference cross-section parameters estimated by the cross-section parameter estimation unit 42, the central axis vector, and the intersection line information estimated by the intersection line estimation unit 44.

[0080] Figures 8A to 8D are diagrams schematically showing the processing when updating the cross-section information using the central axis vector. Figures 8A and 8B show two intersection cross-sections, and the horizontal line 8 in each figure 04, 805 is the intersection line between the reference cross-section before update and the cross-section. And by the process of step S605, at each cross-section, the coordinates of points 810, 811, 812, and 813 are respectively estimated. At this time, first, an angle difference 807 which is the angle formed by the vector connecting point 810 and point 811 and the horizontal line 804, and an angle difference 808 which is the angle formed by the vector connecting point 812 and point 813 and the horizontal line 805 are respectively calculated. Next, the two calculated angle differences are averaged. This average angle difference is the "rotation angle" used for cross-section information update. Finally, the reference cross-section before update is rotated by the rotation angle around the central axis vector. FIG. 8C shows the initial reference cross-section before rotation within the input 3D image 801 space, and FIG. 8D shows the reference cross-section 803 rotated around the central axis vector 830 within the input 3D image 801 space.

[0081] In this way, the update of the reference cross-section parameters is performed considering both the inflow axis of the right ventricle estimated in step S603 and the intersection line estimated based on the cross-section.

[0082] Note that it is also possible to update the cross-section information without performing the process described in this step, by using the method of obtaining an approximate plane passing through the estimated point group coordinates, similar to step S205 in the first embodiment.

[0083] As described above, according to this embodiment, instead of the vertical direction of the reference cross-section before update, the cross-section is calculated using the "central axis" estimated based on the reference cross-section before update. By doing so, the cross-section image group becomes a sliced image reflecting the posture of the right ventricle in the input 3D image, and the variation between cases of the images drawn on the cross-section becomes smaller. Therefore, it becomes easier to perform the inference process of calculating the intersection line from this cross-section, and it becomes possible to perform more appropriate correction of the reference cross-section parameters.

[0084] In the second embodiment, the case where the central axis vector is estimated based on the reference cross-section before update in step S603 has been described as an example. However, the implementation of the present invention is not limited to this, and it is also possible to directly calculate the central axis vector from the input three-dimensional image without passing through the reference cross-section before update. For example, a ring-shaped shape representing the shape of the tricuspid valve annulus is detected from the input three-dimensional image, and after obtaining an approximate plane on which the shape lies, the normal vector of the plane may be used as the central axis vector. In this case, even when the points of the tricuspid valve annulus cannot be estimated well in the reference cross-section before update, it is possible to estimate the central axis vector in consideration of the three-dimensional structure of the right ventricle. Also, based on both the central axis obtained by the method described in step S603 of the second embodiment and the central axis obtained by the above processing, the central axis used in the subsequent processing steps may be calculated. Specifically, it can be obtained as the average of the two central axes, or one of the two central axes may be selected based on the image quality of the reference cross-section before update, etc.

[0085] Also, in the second embodiment, when calculating a plurality of cross-sections in step S604, all the cross-sections were assumed to be in a parallel relationship with each other. However, as long as they do not intersect the reference cross-section before update at the same position, they do not necessarily have to be parallel to each other. That is, the normal vectors of each cross-section may deviate from a parallel relationship with each other within a range of a predetermined value (for example, ±5 degrees). For example, the contrast (the difference between the minimum pixel value and the maximum pixel value) of the calculated cross-section image is measured, and a cross-section image with the maximum contrast may be obtained under the condition that the cross-sections do not intersect each other and the rotation amount of the normal vector is within a predetermined value (for example, ±5 degrees). In this case, since a cross-section image with high contrast and easy to identify the tissue pattern is input to the estimation process in step S605, it is possible to improve the estimation accuracy of the intersection line estimation.

[0086] <Third Embodiment> Next, the third embodiment will be described. In the description of this embodiment, the same components and processes as those in the first and second embodiments are denoted by the same reference numerals, and detailed descriptions thereof are omitted. 。Similar to the first and second embodiments, the image processing apparatus 10 according to the third embodiment estimates parameters (reference cross-section parameters) representing the position and orientation of a reference cross-section for observing the right ventricle, using a three-dimensional image as an input image. In the second embodiment, when calculating the cross-section group, only the information (cross-section parameters and central axis vector) obtained from the reference cross-section before update was used. On the other hand, in this embodiment, information obtained from another cross-section (auxiliary cross-section) derived from the reference cross-section before update is further used to calculate the cross-section image group. By doing so, it becomes possible to reflect anatomical features of the right ventricle that do not appear in the reference cross-section before update, such as the supraventricular crest, in the calculation of the cross-section image group.

[0087] With reference to FIGS. 9A to 9C, the "another cross-section calculated from the reference cross-section before update" in this embodiment will be described. FIG. 9A shows the reference cross-section before update, which is referred to as "plane A" in this embodiment. This reference plane A before update (initial estimated plane A) is the same cross-section as the reference cross-section before update (FIG. 7A) in the second embodiment. Then, the cross-section 902 obtained by rotating the parameters of this initial estimated plane 901 by 90 degrees around the central axis 910 in the input three-dimensional image space is called "plane B". The mode of plane B is schematically shown in FIG. 9B. Compared to plane A that can simultaneously observe the four chambers of the heart, plane B is a cross-section that can observe the position called the "supraventricular crest" where the inflow path and the outflow path join. In this embodiment, the coordinates of the supraventricular crest 920 shown in FIG. 9B are detected from the plane B cross-section image and used in the calculation of the cross-section image group.

[0088] Continuing to refer to FIGS. 9A to 9C, a method for calculating a cross-sectional image group in the present embodiment will be described. It is assumed that a reference cross-section (initial estimated A-plane 901) before update has been calculated in advance. At this time, first, the positions of the apex 921 and two points 925 and 926 of the left and right tricuspid valve annuli are estimated (detected) from the initial estimated A-plane. Then, the midpoint 924 of the two points of the left and right tricuspid valve annuli is calculated. Second, an axis (central axis 910) orthogonal to the axis connecting the two points of the left and right tricuspid valve annuli and passing through the calculated midpoint is calculated. Third, the B-plane cross-sectional image is calculated by rotating the initial estimated A-plane 90 degrees around the calculated central axis, and the position of the supraventricular crest is detected on the B-plane cross-sectional image. Fourth, a point 923 is calculated by vertically dropping the supraventricular crest 920 toward the central axis 910. As shown in FIG. 9C, since the A-plane and the B-plane intersect at the central axis 910, the point 923 dropped in this way is a point located on both the initial estimated A-plane and the B-plane. Fifth, on the initial estimated A-plane, the apex 921 is vertically dropped toward the central axis 910, and the coordinates of the point 922 are calculated. As a result of these processes, three points 922, 923, and 924 exist on the central axis 910. Then, three planes 903, 904, and 905 orthogonal to the central axis and passing through these three points respectively are the cross-sectional group in the present embodiment.

[0089] The configuration and processing of the image processing apparatus according to the present embodiment will be described. The configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the first and second embodiments (FIG. 1). Also, the processing of the image acquisition unit 41, the intersection line estimation unit 44, the cross-sectional information update unit 45, and the display processing unit 51 is the same as the processing content in the second embodiment.

[0090] The cross-sectional parameter estimation unit 42, similar to the second embodiment, estimates a central axis that defines the reference cross-sectional parameters before update and the posture of the right ventricle from the input three-dimensional image acquired by the image acquisition unit 41. In the present embodiment, further, based on another cross-sectional information calculated from the reference cross-section before update, feature information that defines the parameters of some cross-sections of the cross-sectional group is estimated. The details of the processing will be described in detail in the description of steps S602 and S603.

[0091] Similar to the first and second embodiments, the intersection cross-section group acquisition unit 43 acquires a plurality of two-dimensional cross-section images representing a plurality of cross-sections (intersection cross-section group) that intersect with the reference cross-section before update. In this embodiment, it is different from the second embodiment in that the process is performed using not only the reference cross-section before update and the central axis but also the feature information estimated by the cross-section parameter estimation unit 42. The details of the process will be described in detail in the description of step S604.

[0092] The flowchart of the process according to this embodiment is the same as the flowchart (FIG. 6) in the second embodiment. However, the processes in steps S603 and S604 are different from those in the second embodiment. Hereinafter, only the process steps different from the second embodiment will be described.

[0093] (Step S603: Estimation of feature information) In step S603, similar to the second embodiment, the cross-section parameter estimation unit estimates the reference cross-section parameters before update and the central axis that determines the posture of the right ventricle from the input three-dimensional image. In this embodiment, in addition to those, feature position information based on the anatomical features of the right ventricle is estimated from each of another cross-section calculated from the reference cross-section before update and the reference cross-section before update. These feature position information are used to determine the central position of the intersection cross-section group in the subsequent process. Here, the description of the estimation of the initial estimation reference parameters and the central axis, which is a process common to the second embodiment, will be omitted, and only the estimation of the feature position information will be described.

[0094] The process of this step will be described with reference to FIGS. 9A to 9C. By the process common to the second embodiment, two points 925 and 926 of the left and right tricuspid valve rings, their midpoint 924, and the central axis 910 are calculated. First, the initial estimation A plane is rotated 90 degrees around the central axis 910 to obtain the B plane (FIG. 9 Obtain (B). Next, detect the supraventricular crest 920 from the B-plane image and drop it vertically with respect to the central axis 910. As described in the beginning of this embodiment, the point 923 dropped in this way is a point located on both the initially estimated A-plane and B-plane. Next, on the initially estimated A-plane, drop the apex 921 vertically with respect to the central axis 910 and calculate the point 922. As a result of these processes, three points 922, 923, and 924 are calculated on the central axis 910. The estimation of the position of the supraventricular crest 920 is performed based on a CNN.

[0095] The coordinate values of the three points obtained by the above processing, and the reference cross-section parameters and central axis vector before update obtained by the processing common to the second embodiment are the information obtained as the processing result of this step.

[0096] In addition to the above three points, for example, additional feature information points may be calculated from the above three points, such as the midpoint between point 922 and point 924, and the midpoint between point 922 and point 923. By doing so, it is possible to reduce the deviation in the position of the cross-sectional image group set in the subsequent processing that may occur when only the above three points are used.

[0097] Also, in this embodiment, the feature information is estimated on both the reference cross-section (initially estimated A-plane) before update and the B-plane, but it is also possible to be based only on the initially estimated A-plane (that is, not use the supraventricular crest which is information obtained from the B-plane). In this case, since the process of detecting feature points on the B-plane cross-sectional image can be omitted, the calculation cost can be reduced.

[0098] (Step S604: Calculation of cross-sectional image group) In step S604, the cross-section group acquisition unit 43 acquires a cross-sectional image group obtained by cutting the input three-dimensional image in a plurality of cross-sections (cross-section group) that are in an intersecting relationship with the reference cross-section. Here, each of the plurality of cross-sectional images constituting the cross-sectional image group is a two-dimensional image.

[0099] In the present embodiment, the center point of each cross-sectional image group (i.e., the position where the cross-section is penetrated by the central axis vector) is set to the position of the point group estimated in step S603 and dropped onto the central axis vector. That is, the number of point groups estimated in step S603 is the same as the number of cross-sectional image groups calculated in this step. For other processes, they are the same as the processes in step S604 of the second embodiment.

[0100] As described above, according to the present embodiment, when calculating the cross-sectional group, not only the "posture" of the right ventricle determined by the central axis vector but also the "position" of the anatomical feature points of the right ventricle is considered. Therefore, the variation between cases of the cross-sectional image group is suppressed, and it becomes possible to more stably perform the cross-line estimation on the cross-sectional image group.

[0101] <Fourth Embodiment> Next, the fourth embodiment will be described. FIG. 10 is a block diagram showing a configuration example of an image processing system (medical image processing system) including the image processing apparatus of the present embodiment. The image processing system 2 includes an image processing apparatus 100 and a database 22. The image processing apparatus 100 is connected to the database 22 via a network 21 in a communicable state. The network 21 includes, for example, a LAN or a WAN. In the description of the present embodiment, the same components and processes as those in the first to third embodiments are denoted by the same reference numerals, and detailed descriptions thereof are omitted.

[0102] The image processing apparatus 100 according to the fourth embodiment, similar to the first to third embodiments, takes a three-dimensional image as an input image and estimates parameters (reference cross-section parameters) representing the position and orientation of a reference cross-section for observing the right ventricle. In the first to third embodiments, in order to obtain a group of cross-sectional images, the pre-update reference cross-section estimated from the input three-dimensional image was used. In this embodiment, without obtaining the pre-update reference cross-section, a group of cross-sectional images is directly calculated from the input three-dimensional image. By doing so, since the calculation of the cross-sectional images is not affected by the success or failure of the initial estimated cross-section. Even when the pre-update reference cross-section cannot be estimated successfully, such as when the four chambers of the heart are not well depicted, it is possible to execute the reference cross-section estimation.

[0103] Next, each part of the image processing apparatus 100 will be described. The processes performed by the image acquisition unit 101, the intersection line estimation unit 103, and the display processing unit 51 in FIG. 10 are the same as the processes performed by the image acquisition unit 41, the intersection line estimation unit 44, and the display processing unit 51 in FIG. 1, respectively.

[0104] The cross-sectional image group acquisition unit 102 estimates a plurality of cross-sectional groups (cross-sectional image groups) that are expected to have an intersecting relationship with the reference cross-section estimated later based on the input three-dimensional image acquired by the image acquisition unit 101. Different from the first to third embodiments, these cross-sectional image groups are calculated based only on the input three-dimensional image. The details of the process will be described in detail in the description of step S1102.

[0105] The cross-sectional information calculation unit 105 calculates the parameters of the reference cross-section using the input three-dimensional image acquired by the image acquisition unit 101 and the intersection line information estimated by the intersection line estimation unit 103. The details of the process will be described in detail in the description of step S1104.

[0106] Hereinafter, an example of the image processing apparatus 100 in the fourth embodiment will be described using the flowchart of FIG. 11. Here, since the processes of step S1101 and step S1103 are the same as the processes of step S201 and step S204 in FIG. 2, respectively, the description thereof will be omitted.

[0107] (Step S1102: Estimation of Cross-Section Image Group) In step S1102, the cross-section group acquisition unit 102 estimates a plurality of cross-section groups (cross-section image group) that are expected to form an intersection relationship with the reference cross-section estimated in a subsequent stage based on the input three-dimensional image.

[0108] The estimation of the cross-section image group in this embodiment will be described with reference to FIG. 12. First, using the input three-dimensional image 1201 as the input to the learning model, the positions of the center point 1215 of the tricuspid valve annulus and the apex 1216 are detected respectively. As an example, a CNN is adopted as the learning model used for the detection. Next, cross-sections 1202 and 1204 that pass through points 1215 and 1216 and are orthogonal to the vertical axis of the input three-dimensional image, and the cross-section 1203 in the middle of these two cross-sections are calculated respectively. Among the cross-section groups calculated in this way, the two cross-sections, i.e., the cross-section 1204 passing through the center point of the tricuspid valve annulus and the middle cross-section 1203, are the cross-section image group calculated in this step. The cross-section 1202 passing through the apex is excluded from the cross-sections because this cross-section is not the cross-section that divides the right ventricle and the left ventricle into sections, and there is a possibility that the cross-line information cannot be calculated in the process of step S1103 in the subsequent stage. Then, the cross-section images of each cross-section are calculated using the same process as in step S203 of the first embodiment.

[0109] (Step S1104: Calculation of Reference Cross-Section Parameters) In step S1104, the cross-section information calculation unit 105 calculates the parameters of the reference cross-section using the input three-dimensional image acquired by the image acquisition unit 101 and the cross-line information estimated by the cross-line estimation unit 103.

[0110] The process of this step is based on the same process as step S205 in the first embodiment as follows. First, as a premise, by the process of step S1103 in the previous stage, as shown in FIG. 5D, the end points 502, 503, 504, 505 of the cross-line are calculated. At this time, by using the least squares method, which is a known method, an approximate plane passing through these points is calculated to calculate the reference cross-section (plane 520 in FIG. 5D).

[0111] In addition, for a technique of calculating the parameters of a single cross-section from a plurality of point clouds existing in a three-dimensional image, any method other than the method using the least squares method can be used. For example, a CNN may be used, or a method based on PCA adopted in the above-described conventional technique can also be used. When a method based on learning data is used in this way, even if the end points of the intersection lines calculated in step S1103 include outliers, robust parameter calculation can be performed by learning including such outliers.

[0112] As described above, according to the present embodiment, the cross-section image group is directly calculated from the input three-dimensional image without passing through the reference cross-section before update. By doing so, since the cross-section calculation is not affected by the success or failure of the initial estimated cross-section estimation, even when the reference cross-section before update cannot be estimated well, such as when the four chambers of the heart are not well depicted, it is possible to execute the reference cross-section estimation.

[0113] <Other Embodiments> Further, the disclosed technology can take an embodiment as, for example, a system, an apparatus, a method, a program, or a recording medium (storage medium). Specifically, it may be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to an apparatus composed of one device.

[0114] The present invention supplies a program for causing a computer to execute each step of the image processing method according to the above-described embodiment to a system or an apparatus via a network or a storage medium. Such a program is configured to be read and executed by one or more processors in the computer of the system or apparatus. It can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

[0115] The disclosure of the present embodiment includes the following configurations, methods, and programs. (Configuration 1) An image acquisition unit that acquires a three-dimensional image including a subject as an object; An intersection cross-section acquisition unit that acquires information related to a plurality of cross-sections that intersect with a predetermined reference cross-section from the three-dimensional image; Based on the information related to the plurality of cross-sections, the intersection of the plurality of cross-sections and the reference cross-section An intersection line information acquisition unit that acquires intersection line information, which is information related to the intersection lines; A cross-section information acquisition unit that acquires reference cross-section information, which is information related to the reference cross-section, based on the intersection line information; An image processing apparatus, characterized by comprising the above. (Configuration 2) The image processing apparatus further includes a position and orientation acquisition unit that acquires information including at least one of the position and orientation of the subject from the three-dimensional image, wherein the intersection cross-section acquisition unit acquires information related to the plurality of cross-sections based on the information acquired by the position and orientation acquisition unit. The image processing apparatus according to Configuration 1, characterized by the above. (Configuration 3) The image processing apparatus further includes an estimation result acquisition unit that acquires an estimation result of information related to the reference cross-section from the three-dimensional image, wherein the intersection cross-section acquisition unit acquires information related to the plurality of cross-sections based on the estimation result, and the cross-section information acquisition unit acquires the reference cross-section information by updating the estimation result based on the information related to the plurality of cross-sections. The image processing apparatus according to Configuration 1 or 2, characterized by the above. (Configuration 4) The image processing apparatus according to any one of Configurations 1 to 3, characterized in that the plurality of cross-sections intersect the reference cross-section at different positions. (Configuration 5) The image processing apparatus according to Configuration 3, characterized in that the estimation result includes information related to at least one of the position and orientation of the reference cross-section. (Configuration 6) The image processing apparatus according to Configuration 3 or 5, characterized in that the reference cross-sectional information includes information regarding at least one of the position and orientation of the reference cross-section related to the update of the estimation result. (Configuration 7) The image processing apparatus further includes a central axis acquisition unit that acquires information regarding a central axis representing the orientation of the subject in the three-dimensional image, wherein the cross-sectional plane acquisition unit acquires a plurality of cross-sectional planes intersecting the central axis as the plurality of cross-sectional planes based on the information regarding the central axis. The image processing apparatus according to any one of Configurations 1 to 6, characterized in that (Configuration 8) The image processing apparatus according to Configuration 7, characterized in that the cross-sectional information acquisition unit acquires the reference cross-sectional information based on the information regarding the central axis and the plurality of cross-sectional planes. (Configuration 9) The image processing apparatus according to Configuration 7 or 8, characterized in that the central axis is an axis on the reference cross-section. (Configuration 10) The image processing apparatus further includes an estimation result acquisition unit that acquires an estimation result of information regarding the reference cross-section from the three-dimensional image, wherein the cross-sectional plane acquisition unit acquires the plurality of cross-sectional planes based on the estimation result of the information regarding the reference cross-section and feature information acquired from the central axis. The image processing apparatus according to any one of Configurations 7 to 9, characterized in that (Configuration 11) The estimation result acquisition unit further calculates information regarding an auxiliary cross-section based on the estimation result of the information regarding the reference cross-section, wherein the cross-sectional plane acquisition unit further calculates feature information of the subject from the estimation result and the information regarding the auxiliary cross-section, and acquires the plurality of cross-sectional planes based on the feature information. The image processing apparatus according to Configuration 10, characterized in that (Configuration 12) The subject is a human heart, wherein the reference cross-section is at least one of an apical four-chamber view and a short-axis view of the right ventricle. The image processing apparatus according to any one of Configurations 1 to 11, characterized in that (Configuration 13) The image processing apparatus according to any one of Configurations 1 to 12, wherein the three-dimensional image is a three-dimensional ultrasonic image. (Configuration 14) The image processing apparatus according to Configuration 3, wherein the cross-sectional information acquisition unit acquires the estimation result by an inference process based on the three-dimensional image. (Configuration 15) The image processing apparatus according to Configuration 1, wherein the intersecting line information acquisition unit acquires information related to the intersecting line by an inference process based on the three-dimensional image. (Configuration 16) The three-dimensional image is any one of an ultrasonic image obtained by an ultrasonic diagnostic apparatus, an X-ray CT (Computed Tomography) image obtained by an X-ray CT apparatus, and an MRI (Magnetic Resonance Imaging) image obtained by an MRI apparatus. The image processing apparatus according to any one of Configurations 1 to 15, characterized in that. (Configuration 17) The estimation result acquisition unit uses a learning model trained to output a parameter indicating information related to the reference cross-section in the three-dimensional image with the three-dimensional image as an input, and acquires information related to the reference cross-section. The image processing apparatus according to Configuration 3, characterized in that. (Method 1) An image acquisition step of acquiring a three-dimensional image including a subject as a subject, An intersecting cross-section acquisition step of acquiring information related to a plurality of intersecting cross-sections having an intersecting relationship with a predetermined reference cross-section from the three-dimensional image, An intersecting line information acquisition step of acquiring intersecting line information, which is information related to an intersecting line between the plurality of intersecting cross-sections and the reference cross-section, based on the information related to the plurality of intersecting cross-sections, A cross-sectional information acquisition step of acquiring reference cross-sectional information, which is information related to the reference cross-section, based on the intersecting line information, An image processing method, characterized by including. (Program 1) A program for causing a computer to execute each step of the image processing method described in Method 1.

Explanation of Signs

[0116] 10 Image processing apparatus, 41 Image acquisition unit, 43 Intersection cross-section group acquisition unit, 44 Intersection line estimation unit, 45 Cross-section information update unit

Claims

1. an image acquisition unit for acquiring a three-dimensional image including a subject as a subject; an intersecting cross section acquisition unit for acquiring information on a plurality of intersecting cross sections that are in an intersecting relationship with a predetermined reference cross section from the three-dimensional image; a crossing line information acquisition unit that acquires crossing line information, which is information related to a crossing line between the plurality of crossing cross sections and the reference cross section, based on information related to the plurality of crossing cross sections; a cross-section information acquisition unit that acquires reference cross-section information, which is information related to the reference cross-section, based on the intersection line information; 13. An image processing device comprising:

2. a position and orientation acquiring unit for acquiring information including at least one of a position and an orientation of the subject from the three-dimensional image; The intersecting cross section acquisition unit acquires information related to the plurality of intersecting cross sections based on the information acquired by the position and orientation acquisition unit.

2. The image processing device according to claim 1,

3. an estimation result acquisition unit that acquires an estimation result of information related to the reference cross section from the three-dimensional image, the intersecting cross section acquisition unit acquires information relating to the plurality of intersecting cross sections based on the estimation result, The cross-section information acquisition unit acquires the reference cross-section information by updating the estimation result based on information related to the plurality of intersecting cross sections.

2. The image processing device according to claim 1,

4. The image processing apparatus according to claim 1 , wherein the plurality of cross sections intersect with the reference section at different positions from one another.

5. The image processing apparatus according to claim 3 , wherein the estimation result includes information regarding at least one of a position and a posture of the reference cross section.

6. The image processing apparatus according to claim 3 , wherein the reference cross section information includes information on at least one of a position and an orientation of the reference cross section related to updating of the estimation result.

7. a central axis acquiring unit for acquiring information related to a central axis representing a direction of the subject in the three-dimensional image, The intersecting cross section acquisition unit acquires a plurality of cross sections intersecting the central axis as the plurality of intersecting cross sections based on information related to the central axis.

2. The image processing device according to claim 1,

8. The image processing apparatus according to claim 7 , wherein the cross-section information acquisition unit acquires the reference cross-section information based on information related to the central axis and the plurality of intersecting cross sections.

9. The image processing apparatus according to claim 7 , wherein the central axis is an axis on the reference cross section.

10. an estimation result acquisition unit that acquires an estimation result of information related to the reference cross section from the three-dimensional image, The intersecting cross section acquisition unit acquires the plurality of intersecting cross sections based on the estimation result of the information related to the reference cross section and feature information acquired from the central axis.

8. The image processing device according to claim 7,

11. The estimation result acquisition unit further calculates information related to an auxiliary cross section based on the estimation result of the information related to the reference cross section. The intersecting cross section acquisition unit further calculates characteristic information of the subject from the estimation result and information related to the auxiliary cross section, and acquires the plurality of intersecting cross sections based on the characteristic information. The image processing device according to claim 10 .

12. the subject is a human heart, The reference cross section is at least one of an apical four-chamber view and a right ventricular short-axis view.

2. The image processing device according to claim 1,

13. The image processing apparatus according to claim 1 , wherein the three-dimensional image is a three-dimensional ultrasound image.

14. The image processing apparatus according to claim 3 , wherein the cross-sectional information acquisition unit acquires the estimation result by an inference process based on the three-dimensional image.

15. The image processing device according to claim 1 , wherein the intersection line information acquisition unit acquires information relating to the intersection lines by performing an inference process based on the three-dimensional image.

16. The three-dimensional image is any one of an ultrasound image obtained by an ultrasound diagnostic device, an X-ray CT (Computed Tomography) image obtained by an X-ray CT device, and an MRI (Magnetic Resonance Imaging) image obtained by an MRI device.

2. The image processing device according to claim 1,

17. The estimation result acquisition unit acquires information related to the reference cross section using a learning model that is trained to receive the three-dimensional image and output parameters indicating information related to the reference cross section in the three-dimensional image.

4. The image processing device according to claim 3.

18. an image acquiring step of acquiring a three-dimensional image including the subject as a subject; an intersecting cross section acquiring step of acquiring information related to a plurality of intersecting cross sections that are in an intersecting relationship with a predetermined reference cross section from the three-dimensional image; a crossing line information acquiring step of acquiring crossing line information, which is information related to a crossing line between the plurality of crossing cross sections and the reference cross section, based on information related to the plurality of crossing cross sections; a cross-section information acquiring step of acquiring reference cross-section information, which is information related to the reference cross-section, based on the intersection line information; 13. An image processing method comprising:

19. A program for causing a computer to execute each step of the image processing method according to claim 18.

Images