Image processing apparatus, image processing method, and program

Abstract

To provide an image processor for acquiring a reference cross section group to be used for the observation and image processing of a three dimensional image at a higher speed with higher accuracy.SOLUTION: An image processing apparatus includes an image acquisition unit configured to acquire an input image that is a three dimensional image of an object, and a cross section acquisition unit configured to acquire a first cross section group including a plurality of first cross sections that are set to observe the object in the input image and have a predetermined angular relationship with each other, and a second cross section group including a plurality of second cross sections based on a constraint condition based on the predetermined angular relationship.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In the medical field, three-dimensional images acquired by various imaging diagnostic devices (modalities) such as ultrasound imaging diagnostic devices are used for diagnosis. In diagnosis using three-dimensional images, doctors set multiple planes (hereinafter also referred to as reference planes) to be observed in the three-dimensional images, and perform diagnosis on the reference planes or apply various image processing such as measurement or functional evaluation. The multiple reference planes (hereinafter also referred to as a group of reference planes) have predetermined positional and angular relationships with each other. For example, in three-dimensional transesophageal echocardiography, three reference planes that are approximately orthogonal to each other are used.

[0003] As a technique for acquiring a cross section of a 3D image, for example, Patent Document 1 discloses a technique for detecting characteristic points of a search site from a candidate cross section and acquiring an observation cross section using information on the detected characteristic points. Also, Patent Document 2 discloses a technique for adjusting the neck plane of a aneurysm by superimposing different cross-sectional images on two cross-sectional images that are non-orthogonal to each other and displaying them. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2016 / 195110 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-35379 Summary of the Invention [Problem to be solved by the invention]

[0005] The reference cross sections for observing and measuring 3D images can be set manually, for example. That is, an operator such as a doctor identifies which cross sections are the reference cross sections on the 3D images. Setting the reference cross sections manually increases the operator's workload. Furthermore, the manually set reference cross sections may lack consistency and reproducibility. Furthermore, even if the observation cross sections are obtained using information on the feature points of the searched region, it may not be possible to efficiently search for the reference cross sections.

[0006] An object of the present invention is to provide an image processing device that acquires a group of reference cross sections used for observing and processing three-dimensional images at higher speed and with higher accuracy. [Means for solving the problem]

[0007] In order to solve the above problems, the image processing device according to the present invention comprises: an image acquisition unit that acquires an input image that is a three-dimensional image of an object; a cross-section acquisition unit that acquires a first cross-section group consisting of a plurality of first cross-sections that are set for observing the object in the input image and have a predetermined angular relationship with each other, and a second cross-section group consisting of a plurality of second cross-sections based on a constraint condition based on the predetermined angular relationship; The present invention is characterized by comprising: [Effects of the Invention]

[0008] According to the present invention, a group of reference cross sections used for observing and processing three-dimensional images can be acquired at higher speed and with higher accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating a reference cross section. [Figure 2] FIG. 1 is a diagram illustrating an example of the configuration of an image processing system according to a first embodiment. [Figure 3] 10 is a flowchart illustrating a reference cross section acquisition process according to the first embodiment. [Figure 4]10 is a flowchart illustrating a learning model generation process. [Figure 5] 10 is a flowchart illustrating a first cross section acquisition process. [Figure 6] 10 is a flowchart illustrating a second cross section acquisition process. [Figure 7] FIG. 10 is a diagram illustrating an example of the configuration of an image processing system according to a second embodiment. [Figure 8] 10 is a flowchart illustrating a reference cross section acquisition process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the components described in the following embodiments are merely examples, and the technical scope of the present invention is not limited to individual embodiments.

[0011] The image processing device according to the embodiment inputs a three-dimensional image (three-dimensional volume) of a subject (object) and provides a function for acquiring (estimating) two or more types of reference cross sections. The input image to be processed is a medical image, i.e., an image of a subject (such as a human body) captured or generated for the purpose of medical diagnosis, examination, research, etc., and is typically an image acquired by an imaging system called a modality. For example, input images that can be processed include ultrasound images acquired by an ultrasound diagnostic device, X-ray CT images acquired by an X-ray CT device, and MRI images acquired by an MRI device.

[0012] The following embodiment will explain in detail a specific example of the process by which an image processing device acquires a group of reference cross sections, using an example of a three-dimensional image of the cardiac region (three-dimensional transesophageal echocardiogram) captured with a three-dimensional transesophageal ultrasound probe.

[0013] First Embodiment The image processing device according to the first embodiment acquires three reference planes suitable for observing the mitral valve captured in the three-dimensional image from a three-dimensional image captured by a three-dimensional transesophageal probe. The three reference planes are assumed to be approximately orthogonal to each other.

[0014] This embodiment utilizes the property that the reference cross sections are approximately orthogonal to one another. That is, the image processing device first acquires first cross sections (tentative reference cross sections) under the constraint that the three cross sections are orthogonal to one another. Next, the image processing device acquires second cross sections (reference cross sections) corresponding to each first cross section by varying each first cross section so that the first cross sections of the first cross section group (tentative reference cross section group) do not deviate too much from a state in which they are orthogonal to one another.

[0015] Three examples of the second cross-sections will be described using Figure 1. Figure 1 shows a three-dimensional image of the mitral valve taken using a transesophageal probe, cut into cross-sections at the junction between the left ventricle and the left atrium. Figure 1 shows an aortic valve 101 and a mitral valve 102. The mitral valve 102 consists of two membranes: an anterior leaflet 103 and a posterior leaflet 104.

[0016] In Fig. 1, cross section 111 represented by a straight line is a second cross section passing through the center of aortic valve 101 and the center of mitral valve 102, and is called plane A (YZ plane). Cross section 112 represented by a straight line perpendicular to the line representing cross section 111 is a second cross section passing through two locations where the anterior and posterior leaflets join, and is called plane B (XZ plane). Cross section 113 is a second cross section obtained by cutting the junction between the left ventricle and the left atrium, and is called plane C (XY plane). Planes A, B, and C are a second group of cross sections that are roughly orthogonal to one another.

[0017] When the doctor manually sets the secondary planes, each secondary plane is set in orthogonal relation, taking into account the appearance. In this embodiment, the image processing device acquires a plurality of first cross sections that are orthogonal to one another, and corrects the angles of the acquired first cross sections to acquire second cross sections corresponding to the respective first cross sections.

[0018] [Device configuration] The configuration and processing of the image processing device of this embodiment will be described using Fig. 2. Fig. 2 is a block diagram illustrating the configuration of an image processing system (also referred to as a medical image processing system) according to the first embodiment. The image processing system includes an image processing device 10 and a database 22.

[0019] The image processing device 10 is communicably connected to a database 22 via a network 21. The network 21 includes, for example, a local area network (LAN) and a wide area network (WAN). The image processing device 10 may be integrated with the database 22.

[0020] The database 22 stores and manages multiple images and information. The information managed by the database 22 includes 3D images to be processed by the image processing device 10 and learning data (also referred to as teacher data) for generating a learning model (trained model). The learning data is composed of data on multiple cases (learning cases) prepared for learning. Hereinafter, data on individual cases included in the learning data will be referred to as training case data. Each piece of training case data includes a 3D image of each case and information on the correct second cross section in each image. The information managed by the database 22 may include information on a learning model generated from the learning data instead of the learning data.

[0021] The image processing device 10 can acquire data held in the database 22 via the network 21. Note that the information stored in the database 22 or part of the information stored in the database 22 may be stored in an internal memory (ROM 32 or memory unit 34) of the image processing device 10.

[0022] The image processing device 10 includes a communication IF (Interface) 31, a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33, a storage unit 34, an operation unit 35, a display unit 36, and a control unit 37.

[0023] The communication IF 31 (communication unit) is, for example, a LAN card. The communication IF 31 realizes communication between an external device such as the database 22 and the image processing device 10. The ROM 32 is a non-volatile memory that stores various programs and various data. The RAM 33 is a volatile memory that is used as a work memory that temporarily stores programs and data that are being executed.

[0024] The storage unit 34 is, for example, a hard disk drive (HDD) and stores various programs and various data. The operation unit 35 is, for example, a keyboard, a mouse, a touch panel, etc., and receives input of instructions for each block of the image processing device 10 from a user such as a doctor or a medical technician.

[0025] The display unit 36 ​​is a display or the like, and displays various information to the user. The control unit 37 includes one or more processors, such as a CPU (Central Processing Unit), and controls the overall processing of the image processing device 10. The control unit 37 includes a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), The control unit 37 may include a signal processor (Signal Processor), a field-programmable gate array (FPGA), etc. The image acquisition unit 41, the learning model generation unit 42, the first cross-section acquisition unit 43, the second cross-section acquisition unit 44, and the display control unit 45.

[0026] The image acquisition unit 41 acquires an input image to be processed (a three-dimensional image in which a second cross section has not yet been set) from the database 22. The input image to be processed is an image of a subject acquired by various modalities (imaging systems). In this embodiment, the input image is assumed to be a three-dimensional ultrasound image of the heart. The image acquisition unit 41 may acquire the input image directly from the modality. In this case, the image processing device 10 may be implemented as a console for the modality. In this embodiment, an example will be described in which the input image is a three-dimensional ultrasound image, but the input image may be another type of image.

[0027] The learning model generation unit 42 (first generation unit, second generation unit) acquires learning data from the database 22 and constructs a learning model. The learning case data constituting the learning data each includes pixel value information of a 3D image of each case and information on the correct second cross section (hereinafter referred to as correct second cross section information). The learning model generation unit 42 uses the learning data acquired from the database 22 to construct two types of learning models: a learning model for acquiring the first cross section (first learning model) and a learning model for acquiring the second cross section (second learning model).

[0028] The first cross-section acquisition unit 43 acquires a first group of cross sections using a learning model for acquiring the first cross sections. The first group of cross sections includes a plurality of first cross sections that are orthogonal to each other. The learning model for acquiring the first cross sections is a learning model that acquires cross sections from a 3D image under the constraint that the cross sections are orthogonal to each other. This will be described in detail later in the description of step S303.

[0029] The second cross-section acquisition unit 44 acquires a second cross-section group including second cross-sections corresponding to each first cross-section of the first cross-section group using a learning model for second cross-section acquisition. The second cross-section acquisition unit 44 acquires second cross-sections using the first cross-section group acquired by the first cross-section acquisition unit 43 as initial values. The learning model for second cross-section acquisition is a learning model that acquires second cross-sections corresponding to each first cross-section using each first cross-section as an initial value under the constraint that the positional relationship between the second cross-sections does not deviate too much from the orthogonal relationship. This will be described in detail later in the description of step S304.

[0030] The display control unit 45 displays the results acquired by the second cross-section acquisition unit 44 in the image display area of ​​the display unit 36. The display control unit 45 displays the input image and the information on the acquired (estimated) second cross section in the image display area of ​​the display unit 36 ​​in a display format that allows the user to easily view the information.

[0031] Each component of the image processing device 10 functions according to a computer program. For example, the control unit 37 (CPU) uses the RAM 33 as a work area to read and execute a computer program stored in the ROM 32 or the storage unit 34, thereby realizing the function of each component.

[0032] Note that some or all of the functions of the components of the image processing device 10 may be realized by using dedicated circuits. Also, some of the functions of the components of the control unit 37 may be realized by using a cloud computer. For example, the image processing device 10 may be communicably connected to a computing device in a different location via the network 21, and the image processing device 10 and the computing device may transmit and receive data to each other, thereby realizing the functions of the components of the image processing device 10.

[0033] [Second cross section acquisition process] The second cross-section acquisition process will be described with reference to Fig. 3. Fig. 3 is a flowchart illustrating the second cross-section acquisition process according to the first embodiment.

[0034] (Step S301: Acquire / display image) In step S301, when the user issues an instruction to acquire an image via the operation unit 35, the image acquisition unit 41 acquires the input image specified by the user from the database 22 and stores it in the RAM 33. In addition, the display control unit 45 displays the input image acquired from the database 22 in the image display area of ​​the display unit 36.

[0035] Note that any known method may be used to specify the input image. For example, the input image may be specified directly by the user. Alternatively, the input image may be automatically selected based on a predetermined criterion such as the date and time of shooting. For example, the image acquisition unit 41 may select the image with the most recent date and time of shooting from among unviewed images as the input image.

[0036] (Step S302: Generation of learning model) In step S302, the learning model generation unit 42 acquires learning data from the database 22 and executes a learning process using the acquired learning data to construct a learning model as a learning device. The learning data preferably includes case data of images of multiple different patients as training cases to enhance the robustness of the learning model. The learning data may also include frame images in a series of video data of the same patient, and images of the same patient captured at different times.

[0037] The processing of the learning model generation unit 42 in this step will be described in detail with reference to Fig. 4. Fig. 4 is a flowchart illustrating the learning model acquisition processing.

[0038] (Step S3021: Acquire learning data images and correct second cross-sectional information) In step S3021, the learning model generation unit 42 acquires learning data from the database 22. The learning data includes three-dimensional images and correct second cross-section information of each learning case.

[0039] The 3D images of each training case included in the training data consist of three types of information: the number of voxels in each axis direction (width, height, depth), the physical size of each voxel, and an array that stores the pixel values ​​of each voxel.

[0040] The correct second cross section information of each training case included in the training data is information that defines each second cross section of the training case. As described in FIG. 1, three types of cross sections, namely, plane A, plane B, and plane C, are defined as second cross sections in each training case of the training data. The three types of second cross sections are defined by four vectors each having three parameters, for example, as shown in the following equation (1). In other words, the three types of second cross sections are defined by a total of 12 parameters.

number

[0041] In equation (1), C is the intersection of the three second cross sections, that is, a position vector representing the coordinates of a point in space where the three second cross sections intersect with each other. A ~N C represent the normal vectors of the A, B, and C faces, respectively.

[0042] (Step S3022: Orthogonalizing the correct second cross-sectional information) In step S3022, the learning model generation unit 42 orthogonalizes (corrects) the correct second cross section information for each learning case of the learning data acquired in step S3021. Specifically, the learning model generation unit 42 converts information (correct second cross section information) that defines each second cross section of each learning case into parameters that approximate and represent cross sections that are orthogonal to each other. When the second cross sections are orthogonal to each other, the information defining each second cross section can be expressed by a total of six parameters shown in the following equation (2).

number

[0043] In equation (2), C is a position vector representing the coordinates of the intersection of the three second cross sections, just like C in equation (1). Furthermore, R represents the rotation angle in each axial direction from the X-axis, Y-axis, and Z-axis to the normal vector of each corresponding second cross section. The correct second cross section obtained in S3021 can be expressed as three mutually orthogonal second cross sections, as in equation (2), with fewer parameters than in the case of equation (1), by approximating the second cross sections (their normal vectors) so that they are mutually orthogonal.

[0044] The conversion process for orthogonalizing the correct second cross-sectional information in this step will be specifically described. The learning model generation unit 42 can orthogonalize the correct second cross-sectional information by calculating the parameter R in equation (2).

[0045] Here, an example will be described in which the C plane is given priority and the orientation of the C plane does not change before and after the transformation. First, the learning model generation unit 42 calculates the normal vector N C and the normal vector N of surface B B Calculate the cross product of vector N A By the well-known theorem of orthogonality of cross products, we obtain the vector N A ' is a vector N C and vector N B is orthogonal to both

[0046] Next, the learning model generation unit 42 calculates the vector N C and vector N A ' and calculate the cross product of vector N B ' is obtained. Due to the orthogonality of the cross product, vector N B ' is a vector N A ' and vector N C In this way, the learning model generation unit 42 calculates the vector N A ', vector N B ', vector N C We obtain three mutually orthogonal vectors:

[0047] The learning model generation unit 42 is not limited to the case where the C plane is given priority, but may also give priority to the A plane or the B plane to obtain three vectors that are orthogonal to each other. When the A plane is given priority, the learning model generation unit 42 first A and N C N perpendicular to B ' and then N A and N B N orthogonal to ' C Similarly, when the B side is given priority, the learning model generation unit 42 first obtains N B and N A N perpendicular to C ' and then N B and N C N orthogonal to ' A ' to get.

[0048] The learning model generation unit 42 generates a vector N C The angles that the planes make with the Z axis on the XZ plane and the YZ plane are r x and r y In addition, the learning model generation unit 42 acquires the vector N C In the plane defined by B The angle that ' makes with the X axis is r z The learning model generation unit 42 can acquire six parameters representing the three cross sections that are orthogonal to each other by setting the coordinates of the intersection of the three second cross sections before conversion as the position vector C.

[0049] The parameter for orthogonalizing the correct second cross-sectional information is the vector N A ', vector N B ', vector N C It is only necessary to express the attitude formed in space by these three mutually orthogonal vectors, and it is not limited to the above-mentioned R. The orientation (attitude) of the second group of cross sections that are orthogonal to each other may be expressed by a parameter other than R. For example, the orientation of the second group of cross sections that are orthogonal to each other can be expressed using a general method such as expression by a rotation axis and a rotation angle, or expression by a quaternion.

[0050] Furthermore, the learning model generation unit 42 is not limited to the case where the correct second cross section information is orthogonalized by giving priority to the C plane, but may give priority to other cross sections. Furthermore, the learning model generation unit 42 does not give priority to a specific cross section, but may give priority to a cross section that minimizes the change in the overall axis direction before and after the conversion process. For example, the learning model generation unit 42 sets appropriate six parameters in equation (2) as initial values ​​and calculates the angle error between each axis vector of the second group of cross sections. Then, the learning model generation unit 42 uses a known parameter search method such as a gradient method to search for a set of parameters that minimizes the total value of the axis angle error.

[0051] (Step S3023: Construction of a learning model for obtaining the first cross section) In step S3023, the learning model generation unit 42 (first generation unit) constructs a learning model using the learning data obtained in step S3022, which has been orthogonalized with the correct second cross-section information. The learning model for acquiring the first cross-section is a learning device that receives as input a partial 3D image (sample) cut out from the 3D image to be processed, and outputs whether the input sample is appropriate as a sample for identifying the first cross-section group.

[0052] The learning model generation unit 42 constructs a learning model using, for example, Extremely Randomized Tree (ERT), a well-known method derived from the Random Forest algorithm. ERT is a method disclosed in, for example, non-patent document: Geurts et al., "Extremely Randomized Trees," Machine Learning, vol. 36, number 1, pp. 3-42, 2006.

[0053] The specific procedure for constructing a learning model is described below. First, the learning model generation unit 42 generates a sample by cutting out a portion of the 3D image of the learning case data using information on the 3D image of each learning case data included in the learning data and the orthogonalized second cross-sectional parameters (hereinafter referred to as the correct position). Specifically, the learning model generation unit 42 cuts out a region of a predetermined size (partial 3D image) from the 3D image, centered on the position of the position vector C, according to the orientation indicated by the parameter R. The region cut out from the 3D image is, for example, 50 x 50 x 50 voxels.

[0054] The size of the extracted sample (partial 3D image) is desirably large enough to include the object to be observed when acquiring the second cross section (the mitral valve region in the example of FIG. 1). The extracted sample is resized to, for example, 10 x 10 x 10. The learning model generation unit 42 obtains a vector in which pixel values ​​for 10 x 10 x 10 = 1000 voxels are arranged in a single row, and the obtained vector is regarded as the correct sample. On the other hand, a non-correct sample is a sample extracted at a position or orientation that is significantly deviated from the correct position.

[0055] The learning model generation unit 42 extracts multiple (e.g., 500 each) correct samples and non-correct samples from one training case data. While an almost infinite number of non-correct samples can be generated, only one truly correct sample can be obtained from one training case data.

[0056] To obtain multiple correct samples, the learning model generation unit 42 also considers samples that have been slightly shifted in position, orientation, or scale from the correct position (second cross-sectional parameter) as correct samples. For example, the allowable range for correct samples is ±5 pixels for the position, ±5 degrees for the angle, and 0.95 to 1.05 times for the scale, based on the correct position. In other words, correct samples are samples extracted by randomly changing the position, orientation, and scale of the correct position within the allowable range, and non-correct samples are samples extracted outside the allowable range.

[0057] The learning model generation unit 42 extracts correct samples and incorrect samples for each of the learning data orthogonalized in S3022. The learning model generation unit 42 generates, for example, 500 correct samples and 500 incorrect samples for each case in the learning data.

[0058] The learning model generation unit 42 constructs a classifier (learning model) using the generated correct samples and non-correct samples. The learning model generation unit 42 can construct a classifier that distinguishes whether an input unknown sample is a correct sample or a non-correct sample, for example, using ERT.

[0059] Note that the construction of the classifier is not limited to the use of ERT, and any method that estimates parameters based on training data can be applied. For example, the classifier may be constructed by machine learning using a decision tree, a neural network, deep learning, or an SVM (Support Vector Machine).

[0060] Although the learning model has been described as being constructed using pixel values ​​of a 3D image of training case data, it may also be constructed using data obtained by applying predetermined image processing to the 3D image. For example, the learning model generation unit 42 may generate a differential image by differentiating pixel values ​​in a predetermined direction and extract samples from the differential image. Alternatively, the learning model generation unit 42 may extract samples from an image to which image quality improvement processing such as noise reduction or sharpening has been applied.

[0061] (Step S3024: Construction of a learning model for obtaining a second cross section) In step S3024, the learning model generation unit 42 (second generation unit) constructs a learning model using the learning data (before being orthogonalized) obtained in step S3021. As in step S3023, the learning model generation unit 42 can construct a learning model by ERT. The learning model for acquiring the reference cross section is a learning device that receives a cross-sectional image as input and outputs whether the input cross-sectional image is appropriate as a reference cross section, and is generated for each cross section.

[0062] We will explain the difference between the first learning model for cross-section acquisition constructed in step S3023 and the second learning model for cross-section acquisition constructed in step S2024. The first learning model for cross-section acquisition is a learning model for estimating a total of six parameters included in the position vector C and parameter R in equation (2).

[0063] On the other hand, in the learning model for obtaining the second cross section, it is assumed that the position vector C and the parameter R in equation (2) are known by obtaining the first cross section. The learning model for obtaining the second cross section assumes that the position vector C and the normal vector N in equation (1) are known. A ~Normal vector N C This is a learning model for determining a total of 12 parameters.

[0064] In constructing the learning model for acquiring the second cross section, the learning model generation unit 42 firstly adopts the value obtained in the first cross section acquisition as it is as the position vector C of the intersection coordinates. Next, the learning model generation unit 42 uses the vector N A ~Vector N C To estimate (obtain) these three normal vectors, three learning models are used for each second cross section, one for obtaining plane A, one for obtaining plane B, and one for obtaining plane C. The model for obtaining the first cross section is constructed using a 3D image (partial 3D image) as a sample, while the three learning models for obtaining the second cross sections of planes A, B, and C use a 2D image (cross-sectional image) as a sample.

[0065] The procedure for constructing a learning model for the three second cross sections will be described below. First, the learning model generation unit 42 generates samples by cutting out cross-sectional images of planes A, B, and C from the three-dimensional images of the learning case data using the three-dimensional images and second cross-section parameters (correct positions) of each learning case data included in the learning data.

[0066] Specifically, when generating a cross-sectional image sample of plane A, the learning model generation unit 42 calculates a normal vector NA According to the posture, the cross-sectional image is The learning model generation unit 42 cuts out the sample into a size of, for example, 10 x 10 pixels. The learning model generation unit 42 acquires a vector in which pixel values ​​of 10 x 10 = 100 pixels are arranged in a single row as the correct sample.

[0067] The learning model generation unit 42 similarly acquires vectors of correct samples for sides B and C. In addition, the learning model generation unit 42 generates a predetermined number (e.g., 500 each) of correct samples and non-correct samples from one training case data set using the same procedure as in step S3023.

[0068] The method of constructing the classifier is the same as that in step S3023. However, whereas the learning model generation unit 42 generates one learning model in step S3023, it generates three learning models for acquiring the second cross sections of planes A, B, and C in step S3024.

[0069] As in step S3023, the generation of the learning model is not limited to using ERT, and any method that estimates parameters based on learning data can be applied.

[0070] Furthermore, the learning model generation unit 42 may construct a learning model using different methods in step S3023 and step S3024. For example, the learning model generation unit 42 may construct a learning model based on ERT in step S3023, and construct a learning model based on deep learning in step S3024.

[0071] As in step S3023, the learning model generation unit 42 may extract samples from images to which any image processing, such as differential processing for differentiating pixel values, noise reduction, or sharpening, has been applied, rather than from pixel values ​​of the 3D image of the learning case data. The learning model generation unit 42 may use images to which different image processing has been applied in steps S3023 and S3024. For example, the learning model generation unit 42 may construct a learning model using a differential image in step S3023, and construct a learning model using a sharpened image in step S3024.

[0072] The learning model generation unit 42 thus executes the learning model generation process of step S302. Note that the learning model generation process of step S302 is a process independent of the second cross-section acquisition process executed by the image processing device 10 on the input image. Therefore, the process of step S302 can be performed in advance. The generated learning model is stored in a storage device such as the database 22 or the storage unit 34.

[0073] If a learning model has been generated in advance, in step S302, the learning model generation unit 42 reads the learning model from the storage device and stores it in RAM 33. By generating the learning model in advance, the processing time for the second cross-section acquisition process for the input image is shortened. Note that the learning model generation process may be executed by a device other than the image processing device 10 according to the procedure shown in step S302.

[0074] (Step S303: Obtaining the first cross section group) In step S303, the first cross-section acquisition unit 43 acquires a group of first cross sections using the input image acquired in step S301 and the learning model for first cross-section acquisition generated in step S3023. The first cross-section acquisition unit 43 calculates the positions and orientations of three mutually orthogonal first cross sections. That is, the first cross-section acquisition unit 43 calculates a total of six parameters shown in equation (2) for the input image.

[0075] The first cross-section acquisition unit 43 acquires the first cross-section group using ERT, which is a known classifier. When an unknown sample (partial 3D image) cut out from an input image is input, the classifier classifies whether the input sample is an appropriate sample (correct sample) for the first cross-section group or an inappropriate sample (incorrect sample) for the first cross-section group. The first cross-section acquisition unit 43 acquires the first cross-section group based on the sample determined to be correct by the classifier.

[0076] The processing of the first cross-section acquisition unit 43 in this step will be described in detail with reference to Fig. 5. Fig. 5 is a flowchart illustrating the first cross-section acquisition processing.

[0077] In step S3031, the first cross-section acquisition unit 43 determines whether image processing was applied when the learning model was constructed. If image processing was applied when the learning model was constructed, the process proceeds to S3032. If image processing was not applied when the learning model was constructed, the process proceeds to S3033.

[0078] In step S3032, the first cross-section acquisition unit 43 applies the same image processing to the input image as that applied when the learning model was constructed. For example, if the learning model is constructed using differential images, the first cross-section acquisition unit 43 also applies differential processing to the input image. An image that has undergone the same image processing as that applied when the learning model was constructed is called a preprocessed image.

[0079] In step S3033, the first cross-section acquisition unit 43 randomly varies parameters within a predetermined range from the input image or the preprocessed image to extract samples (partial 3D images). The predetermined range can be, for example, ±100 voxels in each axial direction and ±45 degrees in each axial direction, based on the center of the input image (preprocessed image). Specifically, the extraction of samples is performed using a method similar to that used in constructing the learning model for first cross-section acquisition (step S3023). Each sample extracted here represents a candidate (hypothesis) for the first cross-section group defined by the parameters (position and orientation) used for extraction. The first cross-section acquisition unit 43 inputs the extracted samples to a classifier for first cross-section acquisition and determines whether the input sample is a correct sample or an incorrect sample. The series of processes of extracting and classifying samples is performed a predetermined number of times (for example, 10,000 times).

[0080] In step S3034, the first cross section acquisition unit 43 acquires a first cross section group from samples determined to be correct (candidates for the first cross section group). If there is one sample determined to be correct, the first cross section acquisition unit 43 outputs the position and orientation of the sample as parameters of the first cross section group (hereinafter also referred to as first cross section parameters). On the other hand, if there are multiple samples determined to be correct, the first cross section acquisition unit 43 outputs the average position and average angle of the samples determined to be correct as parameters of the first cross section group. The acquired first cross section parameters are the position vector Cc and the parameter Rc.

[0081] If no sample determined to be correct is found, the first cross-section acquisition unit 43 may display a dialog box on the display unit 36 ​​to notify the user that the first cross-section group has not been found.

[0082] Furthermore, when there are multiple samples determined to be correct, the first cross-section acquisition unit 43 can output the average position and average angle as the output of this step, but this is not limited to this. For example, the first cross-section acquisition unit 43 may apply known clustering such as the K-nearest neighbor method to the group of samples determined to be correct, and calculate the average position and average angle using samples included in a cluster with a large number of samples.

[0083] The first cross-section acquisition unit 43 also acquires the contrast of pixel values ​​of the samples determined to be correct. Alternatively, the first cross-section acquisition unit 43 may select samples whose average position and average angle are higher than a threshold value, and calculate the average position and average angle. In this way, the first cross-section acquisition unit 43 can select samples that satisfy a predetermined criterion from the samples determined to be correct, and calculate the average position and average angle.

[0084] Furthermore, if no sample determined to be correct is found, the first cross-section acquisition unit 43 may continue processing by outputting appropriate parameters (first cross-section) of this step, rather than informing the user that the first cross-section was not found. The appropriate parameters may be, for example, the average position and average angle in the learning data of the correct samples used in constructing the learning model for acquiring the first cross-section.

[0085] Alternatively, the first cross-section acquisition unit 43 may calculate three position parameters, fix the calculated position parameters, and then calculate three angle parameters. Conversely, the first cross-section acquisition unit 43 may calculate three angle parameters, fix the calculated angle parameters, and then calculate three position parameters.

[0086] Furthermore, the acquisition process of the first group of cross sections in this step may be performed by a known method for estimating the position and orientation of an object in a 3D image, by regarding the process as a problem of obtaining a Cartesian coordinate system that defines the position and orientation of the mitral valve, which is the object of observation. For example, a 3D image template of the mitral valve may be created, and the position and orientation of the mitral valve may be estimated by a template matching method.

[0087] (Step S304: Obtaining the second cross section group) In step S304, the second cross-section acquisition unit 44 acquires each second cross-section using the input image acquired in step S301, the first cross-section parameters Cc and Rc calculated in step S303, and the second cross-section acquisition learning model generated in step S3024.

[0088] For example, the second cross-section acquisition unit 44 can acquire second cross sections by calculating (correcting) the orientation of each second cross section represented by a total of 12 parameters shown in Equation (1) for the input image. The second cross-section acquisition unit 44 acquires a group of second cross sections using an ERT classifier, similar to step S303.

[0089] The processing of the second cross-section acquisition unit 44 in this step will be described in detail with reference to Fig. 6. Fig. 6 is a flowchart illustrating the second cross-section acquisition processing.

[0090] In S3041, the second cross section acquisition unit 44 calculates the angle parameter Rc among the first cross section parameters by calculating the normal vector N A c, N B c, N C The second cross section acquisition unit 44 can acquire the normal vectors of each of the first cross sections by rotating the X-axis, Y-axis, and Z-axis vectors in the input image based on the angle parameter Rc.

[0091] In step S3042, the second cross-section acquisition unit 44 determines whether image processing was applied when the learning model was constructed. If image processing was applied when the learning model was constructed, the process proceeds to S3043. If image processing was not applied when the learning model was constructed, the process proceeds to S3044.

[0092] In step S3043, the second cross-section acquisition unit 44 applies the same image processing to the input image as that applied when the learning model was constructed. For example, if the learning model is constructed using differential images, the second cross-section acquisition unit 44 also applies differential processing to the input image. The image to which the same image processing as that applied when the learning model was constructed is applied is The image is referred to as the preprocessed image.

[0093] In step S3044, the second cross-section acquisition unit 44 extracts samples of each second cross-section from the input image or the preprocessed image based on each first cross-section. Unlike step S3033, the samples extracted in this step are two-dimensional cross-sectional images. The second cross-section acquisition unit 44 extracts samples of each of planes A, B, and C while fixing the position vector Cc.

[0094] The second cross section acquisition unit 44 acquires the normal vector N A Within a predetermined angle range centered on c (i.e., the normal vector N A c as an initial value) and defines it as a sample of plane A. Similarly, the second cross-section acquisition unit 44 cuts out samples of plane B and plane C. Each sample cut out here represents a candidate (hypothesis) for each second cross-section defined by the parameters (posture) used for cutting out. The sample of plane A is defined by the position C and normal vector N A If it is cut out based on c, it is not necessarily N A It does not have to be cut out with c as the center (i.e., as the initial value). For example, when a sample of a surface A is given, the position C and the normal vector N A A cost function is defined that is calculated by the weighted sum of the difference with respect to c and an image quality evaluation index (for example, luminance contrast). Then, samples of side A are randomly extracted from the input image, and the sample of side A for which the cost function value is below a certain level can be adopted.

[0095] When cutting out samples of planes A, B, and C, the second cross-section acquisition unit 44 cuts out the two-dimensional cross-sectional samples so that they do not deviate too much from the orthogonal relationship with each other. For example, the second cross-section acquisition unit 44 can cut out the samples so that they do not deviate too much from the orthogonal relationship by setting the predetermined angle range for cutting out the samples to ±5 degrees centered on each normal vector.

[0096] Next, another specific example of a method for cutting out two-dimensional cross-sectional samples of each plane so that they do not deviate too much from the mutually orthogonal relationship will be described. First, after cutting out the samples of plane A, the second cross-section acquisition unit 44 discards samples that deviate by a predetermined angle (for example, ±10 degrees) or more from the average value of the normal vectors of each sample of plane A. Note that the second cross-section acquisition unit 44 does not use the average value of the normal vectors of each sample of plane A, but rather uses the normal vector N A Alternatively, samples that deviate from the target by more than a predetermined angle may be discarded.

[0097] Next, the second cross-section acquisition unit 44 cuts out a sample of surface B and calculates the angle between the average value of the normal vectors of surface A and the normal vector of the cut-out sample of surface B. If the normal vector of the cut-out surface B deviates from the orthogonal (90 degree) relationship with the average value of the normal vectors of surface A by a predetermined angle (for example, ±10 degrees) or more, the second cross-section acquisition unit 44 discards the sample of surface B.

[0098] Similarly, the second cross-section acquisition unit 44 cuts out a sample of plane C and calculates the angle between the average value of the normal vectors of plane A and plane B and the normal vector of the cut-out sample of plane C. If the normal vector of the cut-out sample of plane C deviates from the orthogonal (90-degree) relationship with the average value of the normal vectors of plane A or the average value of the normal vectors of plane B by more than a predetermined angle (for example, ±10 degrees), the second cross-section acquisition unit 44 discards the sample of plane C.

[0099] In this way, the second cross section acquisition unit 44 acquires a set of samples in which the deviations of the A, B, and C planes from the orthogonal relationship are within a predetermined angle range as candidates for the second cross section group. Note that in the above example, the second cross section acquisition unit 44 determines whether the B and C planes deviate from the orthogonal relationship using the A plane as a reference, but it may also determine whether the other two planes deviate from the orthogonal relationship using the B or C plane as a reference.

[0100] The second cross section acquisition unit 44 inputs a set of samples acquired as candidates for the second cross section group to a second cross section acquisition classifier. The input sample set is determined to be a set of samples appropriate for the second cross section group (correct sample set) or an inappropriate set of samples (incorrect sample set). To this end, the second cross section acquisition unit 44 first inputs the samples of the A, B, and C surfaces to the second cross section classifiers for the A, B, and C surfaces, respectively, and determines whether they are correct or incorrect samples. The second cross section acquisition unit 44 then acquires, as a correct sample set, a set of samples (candidates for the second cross section group) in which the samples of the A, B, and C surfaces are all determined to be correct samples by the respective classifiers. As in step S3033, the process of S3044 (determining whether a sample set is correct or incorrect) is executed a predetermined number of times (e.g., 10,000 times).

[0101] In step S3045, the second cross section acquisition unit 44 acquires a second cross section group to be output from the set of samples determined to be correct (candidates for the second cross section group). If there is one set of samples determined to be correct, the second cross section acquisition unit 44 outputs parameters of the second cross section represented by the set of samples (hereinafter also referred to as second cross section parameters). On the other hand, if there are multiple sets of samples determined to be correct, the second cross section acquisition unit 44 outputs the average normal vectors of the samples of each cross section in the multiple sets of correct samples as the respective second cross section parameters.

[0102] If a sample determined to be correct is not found, the second cross-section acquisition unit 44 may display a dialog box on the display unit 36 ​​to notify the user that the second cross-section has not been found.

[0103] Also, similar to step S303, the second cross section acquisition unit 44 may use the average normal vector of samples included in a cluster with a large number of samples as a result of clustering. Also, the second cross section acquisition unit 44 may use samples selected based on criteria such as the brightness contrast of the image. Furthermore, if no sample determined to be correct is found, the second cross section acquisition unit 44 may use the average normal vector of the first cross section (i.e., the position vector Cc and the normal vector N A c, N B c, N C c) may be output as is.

[0104] (Step S305: Display of second cross section acquisition result) In step S305, the display control unit 45 displays cross-sectional images obtained by cutting out the input image at each second cross section obtained in step S304 in the image display area of ​​the display unit 36. In addition to the cross-sectional images, the display control unit 45 may also display images obtained by converting the three-dimensional image that is the input image into a two-dimensional image using a known display method such as surface rendering or volume rendering.

[0105] When displaying a cross-sectional image of a certain second cross-section, the display control unit 45 may draw a line representing another second cross-section intersecting the second cross-section so as to be superimposed on the cross-sectional image, as exemplified in Fig. 1. Fig. 1 shows an example in which lines representing plane A (cross-section 111) and plane B (cross-section 112) are drawn on an image of plane C (cross-section 113).

[0106] The display control unit 45 may further display information on the degree of deviation, indicating how much the second cross section represented by a straight line deviates from an angle perpendicular to the displayed cross section. For example, the display control unit 45 may display text information such as "10 degrees deviation from perpendicular" near the line representing the second cross section, or may express the degree of deviation by a display mode such as the color or thickness of the line. By displaying the degree of deviation of the second cross section, the user can easily observe the degree of mutual orthogonality maintained between the second cross sections in the second cross section acquisition results.

[0107] If the purpose is to analyze or measure a three-dimensional image, the process of step S305 can be omitted. In this case, the image processing device 10 stores the acquired second cross-sectional information in a storage device, and ends the process shown in FIG.

[0108] When multiple correct samples are obtained in step S303, the first cross section acquisition unit 43 limits the parameters to one set of first cross section groups by averaging them, but this is not limited to this. The first cross section acquisition unit 43 may directly transfer the parameters of multiple sets of correct samples (first cross section groups) to the second cross section acquisition unit 44. The second cross section acquisition unit 44 executes the process of step S304 as many times as the number of first cross section groups transferred. The second cross section acquisition unit 44 can acquire one set of second cross section groups by averaging multiple second cross sections acquired based on each first cross section group. By acquiring the second cross section group based on multiple sets of first cross section groups, the risk of falling into an inappropriate local solution is reduced.

[0109] In the first embodiment, the second group of cross sections is acquired so as not to deviate too much from the angular relationship in which the second cross sections are orthogonal to each other. That is, the image processing device 10 restricts the search range for a solution so that the angular relationship between the second cross sections deviates from the orthogonal relationship within a predetermined angle range. This allows the image processing device 10 to acquire the second group of cross sections used for observing or processing three-dimensional images at high speed and with high accuracy.

[0110] (Variation 1) The first embodiment illustrates an example of acquiring a group of second cross sections in which the second cross sections are in a roughly orthogonal angular relationship. In contrast, in the first modification, the angular relationship between the acquired second cross sections does not have to be an orthogonal relationship. In the first modification, the angular relationship between the second cross sections can be a predetermined angular relationship that is set in advance. Below, we will explain the processing that differs from the first embodiment among the processing of each step of the second cross section acquisition processing in FIG. 3 and the learning model generation processing in FIG. 3.

[0111] An example of an angular relationship between second cross sections other than orthogonal is when three types of second cross sections, namely, a four-chamber view (A4C), a two-chamber view (A2C), and a three-chamber view (A3C), are obtained from three-dimensional images obtained by three-dimensional transsternal echocardiography. The four-chamber view (A4C) and the two-chamber view (A2C) are in a roughly orthogonal relationship. However, the three-chamber view (A3C) is a cross section rotated approximately 50 degrees relative to the four-chamber view (A4C), and its angular relationship with the four-chamber view (A4C) is not orthogonal.

[0112] In the first modification, in the orthogonalization of the correct second cross-sectional information in step S3022, the cross-section defining the three-chamber view (A3C) is aligned so as to maintain a 50-degree angle with the cross-section of the four-chamber view (A4C) rather than being orthogonal to the other two cross-sections. The cross-section of the three-chamber view (A3C) can be uniquely represented by the six parameters shown in equation (2) even if the angular relationship with the cross-section of the four-chamber view (A4C) is not orthogonal.

[0113] In steps S3023 and S303, when generating samples by randomly varying parameters, samples in which the cross section of the three-chamber view (A3C) deviates from the cross section of the four-chamber view (A4C) by more than a predetermined angle from 50 degrees are rejected from both correct and incorrect samples.

[0114] In step S304, when extracting samples of each cross section, samples of the two-chamber view (A2C) that deviate from the orthogonal cross section of the four-chamber view (A4C) by a predetermined angle or more are discarded from the candidates for the second cross section. Also, samples of the three-chamber view (A3C) that deviate from 50 degrees from the four-chamber view (A4C) by a predetermined angle or more are discarded from the candidates for the second cross section.

[0115] In the first modification, even when the second cross sections are not in an angular relationship that is approximately orthogonal to each other, the image processing device The apparatus 10 can acquire a second group of cross sections used for three-dimensional image observation or image processing at high speed and with high accuracy.

[0116] (Variation 2) The first embodiment illustrates an example in which the image processing device 10 automatically acquires a first group of cross sections and a second group of cross sections. In contrast, in Modification 2, the image processing device 10 receives input of a first group of cross sections from a user such as a doctor, and acquires a second group of cross sections based on the input first group of cross sections. Below, we will explain the processing of each step of the second cross section acquisition processing in FIG. 3 and the learning model generation processing in FIG. 3 that differs from the first embodiment.

[0117] In Modification 2, the first cross-section group is manually input by the user. In the process of acquiring the first cross-section group in step S303, the display control unit 45 displays a user interface (UI) for manually inputting the first cross-section on the display unit 36. The user inputs the first cross-section through the UI for inputting the first cross-section via the operation unit 35.

[0118] The UI for inputting the first cross sections may be any UI that allows the user to specify three mutually orthogonal first cross sections. For example, the display control unit 45 displays cross-sectional images obtained by cutting a three-dimensional image along three planes, namely, the YZ plane, the XZ plane, and the XY plane, in a tiled pattern. The display control unit 45 displays a cross-shaped figure on each cross-sectional image that can be moved or rotated by operating the mouse.

[0119] For example, in Figure 1, the cross displayed on the YZ plane represents the intersection line where surfaces B and C intersect with the YZ plane. Similarly, the cross displayed on the XZ plane represents the intersection line where surfaces A and C intersect with the XZ plane, and the cross displayed on the XY plane represents the intersection line where surfaces A and B intersect with the XY plane.

[0120] The user can input the first cross section by moving or rotating the crosshair on each cross section image with the mouse. When the position and angle of the crosshair on one of the cross section images is changed, the display control unit 45 reflects the change in the other two cross section images. By having the display control unit 45 accept the input of the first cross section, the user can intuitively manually input the first cross section group while maintaining the orthogonality constraint.

[0121] (Variation 3) In the first embodiment, when a learning model for acquiring the second cross sections is constructed in step S3024 and the second cross section group is acquired in step S304, the second cross section acquisition unit 44 extracts and uses a two-dimensional cross section image from the input image. In contrast, in Modification 3, the second cross section acquisition unit 44 may extract and use a three-dimensional image (partial three-dimensional image). Below, we will explain the processing that differs from the first embodiment among the processing of each step of the second cross section acquisition processing in FIG. 3 and the learning model generation processing in FIG. 3.

[0122] In steps S3024 and S304, the samples cut out by the second cross section acquisition unit 44 are not two-dimensional images (e.g., 10x10 pixels) but three-dimensional images (e.g., 10x10x10 pixels). In addition, in step S304, the process of determining whether the A, B, and C planes are approximately orthogonal to one another is a process of calculating and determining the angles between the normal vectors used to cut out each three-dimensional image sample. In other words, the second cross section acquisition unit 44 discards samples whose normal vectors deviate by a predetermined angle or more from the mutually orthogonal relationship.

[0123] In variant example 3, even when the second group of cross sections is acquired using partial three-dimensional image samples, the image processing device 10 can acquire the second group of cross sections quickly and with higher accuracy, as in the case where two-dimensional image samples are used.

[0124] Second Embodiment The image processing device 10 according to the second embodiment, like the first embodiment, acquires (estimates) a predetermined second cross section from a three-dimensional image (input image) to be processed. The first embodiment is an embodiment in which, after acquiring a first group of cross sections, a second cross section is acquired using the acquired first group of cross sections as an initial position. In contrast, the second embodiment is an embodiment in which the quality of the first group of cross sections is evaluated, and if a predetermined condition is met, the acquired first cross section is output as the second cross section without executing a second cross section acquisition process.

[0125] [Device configuration] The configuration and processing of the image processing device of this embodiment will be described using Fig. 7. Fig. 7 is a block diagram illustrating the configuration of an image processing system according to the second embodiment. Components similar to those in the first embodiment are given the same reference numerals as in Fig. 2 and will not be described again. Differences from the first embodiment will be described below.

[0126] The image processing device 10 includes a determination unit 51 in addition to the configuration shown in Fig. 2. The determination unit 51 determines whether the first cross-section group acquired by the first cross-section acquisition unit 43 is appropriate as the second cross-section group. Details of the determination method will be described later in the description of step S804 in Fig. 8.

[0127] If the acquired first cross-section group is determined to be appropriate as the second cross-section group, the second cross-section acquisition process is not executed. The display control unit 45 regards the first cross-section group as the second cross-section group and displays it in the image display area of ​​the display unit 36, as in the first embodiment.

[0128] [Second cross section acquisition process] The second cross-section acquisition process will be described using Fig. 8. Fig. 8 is a flowchart illustrating the second cross-section acquisition process according to the second embodiment. The processes of steps S801 to S803 and S805 in Fig. 8 are the same as the processes of S301 to S303 and S304 in Fig. 3, and therefore descriptions thereof will be omitted. Differences from the first embodiment will be described below.

[0129] (Step S804: Determine whether to acquire the second cross section) In step S804, the determination unit 51 evaluates the first cross-section group calculated in step S803 and determines whether or not to perform second cross-section acquisition. If the first cross-section group acquired in step S803 satisfies a predetermined condition, the determination unit 51 determines to perform second cross-section acquisition, and if the first cross-section group does not satisfy the predetermined condition, the determination unit 51 can determine not to perform second cross-section acquisition. The first cross-section group satisfies the predetermined condition when the first cross-section group is appropriate as the second cross-section group. If the image processing device 10 determines to perform second cross-section acquisition, the process proceeds to step S805. If the image processing device 10 determines not to perform second cross-section acquisition, the process proceeds to step S806.

[0130] A method for determining whether or not to acquire a reference cross section, i.e., whether or not the first cross section group satisfies a predetermined condition, will be described. The determination unit 51 uses principal component analysis (PCA), a well-known statistical process, to construct a model (subspace) that captures the statistical trends of pixel values ​​of cross section images of each of the A, B, and C planes. The model is constructed by generating cross section images by cutting the 3D images with the correct second cross section information for all learning case data, and then performing principal component analysis on the generated cross section images.

[0131] The determination unit 51 re-expresses the first cross section obtained in step S803 using a corresponding model. For example, the first cross section of plane A is re-expressed using the model of plane A. The determination unit 51 calculates the error (reconstruction error) between the first cross section image and the re-expressed image. First cross section If the first cross section is close to the average tendency of the corresponding cross section (for example, plane A), the reconstruction error by the model will be small. Conversely, if the first cross section is far from the average tendency of the corresponding cross section, the reconstruction error will be large. By calculating the reconstruction error, the determination unit 51 can quantify "how likely the first cross section is to be the corresponding cross section."

[0132] If the reconstruction error for each of the three cross sections is smaller than a predetermined threshold, it is determined that "the acquisition of the second cross section is not to be performed." Conversely, if the reconstruction error is equal to or greater than a predetermined value, it is determined that "the acquisition of the second cross section is to be performed." In this case, the first cross section group can be considered to satisfy the predetermined condition when the reconstruction error is smaller than a predetermined threshold. The predetermined threshold for evaluating the reconstruction error can be, for example, the average value + standard deviation value when a similar reconstruction error is calculated for the training data.

[0133] The method for constructing a model that captures the statistical tendency of pixel values ​​of cross-sectional images is not limited to the method using PCA. The determination unit 51 can use any model as long as it quantifies the "probability that an input image is a predetermined type of image." For example, the determination unit 51 may construct and use a model that determines the probability that an input image belongs to a predetermined class by deep learning. In this case, the first cross-sectional group may satisfy the predetermined condition when the probability that the first cross-sectional group belongs to the predetermined class is greater than a threshold.

[0134] Alternatively, the determination unit 51 may make a simple determination not based on a model, such as determining not to acquire the second cross section when the brightness contrast of the cross-sectional image is equal to or greater than a predetermined threshold. The predetermined threshold may be, for example, the average brightness contrast of the cross-sectional images of the training data. In this case, the first cross-section group may satisfy the predetermined condition when the brightness contrast of the first cross-section group is equal to or greater than a predetermined threshold.

[0135] Furthermore, a model for determining whether or not to acquire the second cross section can be constructed in advance. The model may be constructed by a device other than the image processing device 10. The constructed model is stored in the database 22 or the storage unit 34. In step S804, the determination unit 51 reads the constructed model into the RAM 33 and determines whether or not to acquire the second cross section.

[0136] Although the determination in step S804 has been described using pixel values ​​of the cross-sectional image, the determination unit 51 may make the determination using data obtained by applying predetermined image processing to the cross-sectional image. For example, the determination unit 51 may extract a sample from a differentiated image obtained by differentiating pixel values ​​in a predetermined direction, or from an image to which image quality improvement processing such as noise reduction and sharpening has been applied.

[0137] By applying predetermined image processing to the cross-sectional images, the determination unit 51 can make a more accurate determination than when making a determination using pixel values. Furthermore, the determination unit 51 is not limited to an example in which only cross-sectional images are used, and may perform a similar determination process on a three-dimensional image including regions before and after the cross section, i.e., a determination process by quantifying the probability.

[0138] (Step S806: Display of second cross section acquisition result) In step S806, the display control unit 45 displays, in the image display area, cross-sectional images obtained by cutting out the input image at the second cross-section group obtained in step S805. When the acquisition of the second cross sections is executed (when step S805 is executed), the display control unit 45 performs display using the acquisition results of the second cross sections, similar to the first embodiment.

[0139] If it is determined in step S804 that "acquisition of the second cross section is not executed," the display control unit 45 regards the first cross section, which is a cross section orthogonal to each other, as the second cross section and displays it. The display processing is the same as step S305 in the first embodiment.

[0140] When the first cross-section group acquired in step S803 is displayed as the second cross-sections, the display control unit 45 may notify the user that the second cross-section acquisition process has not been executed. For example, the display control unit 45 may display text information to the effect that the second cross-section acquisition process has not been executed on the display unit. Furthermore, the display control unit 45 may indicate that the mutually orthogonal first cross-sections are being displayed as the second cross-sections by displaying a mathematical symbol representing a right angle near the intersection of the straight lines representing the cross-sections on the display area.

[0141] According to the second embodiment described above, if it is determined that the first cross-section group acquired in step S803 can be used as the second cross-section, the acquisition of the second cross-section is omitted, thereby enabling the image processing device 10 to shorten the processing time.

[0142] Third Embodiment The image processing device 10 according to the third embodiment acquires (estimates) two or more predetermined second cross sections from a three-dimensional image, similar to the first and second embodiments. In the second embodiment, it is determined whether or not to perform second cross section calculation for a group of acquired first cross sections. In contrast, in the third embodiment, it is determined whether or not to perform second cross section acquisition for each cross section. For example, the image processing device 10 according to the third embodiment can determine not to perform second cross section acquisition for plane A but to use the first cross section as the second cross section, but to perform second cross section acquisition for planes B and C.

[0143] The configuration of the image processing device 10 according to the third embodiment is similar to the configuration of the second embodiment shown in Fig. 7. However, the process by the determination unit 51 differs from that of the second embodiment in that it determines whether or not to perform second cross-section acquisition for each cross-section.

[0144] The second cross section acquisition process according to the third embodiment is similar to the flowchart of the second embodiment shown in Fig. 8. However, the processes of steps S804 and S805 differ from those of the second embodiment. The differences will be described below.

[0145] (Step S804: Determine whether to acquire the second cross section) In step S804, the determination unit 51 evaluates the first cross-section group calculated in step S803. The determination unit 51 determines whether or not to acquire a second cross-section for each of the cross-sections A, B, and C.

[0146] If it is determined that "second cross section acquisition is not to be performed" for any cross section, the image processing device 10 proceeds to step S806. If it is determined that "second cross section acquisition is to be performed" for any cross section, the image processing device 10 proceeds to step S805.

[0147] The determination method for each cross section will be explained. The determination unit 51 calculates the reconstruction error for each cross section using the same procedure as in step S804. The determination unit 51 determines that "second cross section acquisition will not be performed" for a cross section whose reconstruction error is below a predetermined threshold. Conversely, the determination unit 51 determines that "second cross section acquisition will be performed" for a cross section whose reconstruction error is equal to or greater than the predetermined threshold, as in the first embodiment.

[0148] The predetermined threshold for evaluating the reconstruction error can be, for example, the average value plus the standard deviation value when the same reconstruction error is calculated for the training data. The information on the determination result in step S804 is used in the processing in step S805.

[0149] (Step S805: Obtaining the second cross section group) In step S805, the second cross-section acquisition unit 44 executes the second cross-section acquisition. The second cross section acquisition unit 44 acquires each second cross section using the input image, the first cross section parameters Cc and Rc, the second cross section acquisition learning model, and the determination information calculated in step S804.

[0150] The process of step S804 shares some of the same processing as step S304 in the first embodiment. The process of step S804 will be described below, clarifying the commonalities and differences with step S304.

[0151] First, the second cross section acquisition unit 44 calculates the angle parameter Rc among the first cross section parameters by calculating the normal vector N A c, N B c, N C c. The first process in step S804 is the same as the first process in step S304.

[0152] Second, the second cross-section acquisition unit 44 cuts out samples of each second cross-section from the input image. If there is a cross-section among the three cross-sections, planes A, B, and C, for which it is determined in step S804 that "second cross-section acquisition will not be performed," the second cross-section acquisition unit 44 does not change the normal vectors corresponding to the cross-sections for which second cross-section acquisition will not be performed. As in the second process in step S304, the normal vectors corresponding to the other cross-sections are calculated so that the normal vectors of each cross-section maintain an angular relationship within a predetermined angle range from the orthogonal relationship. The second cross-section acquisition unit 44 acquires the samples cut out based on the normal vectors of each cross-section as candidates for the second cross-section.

[0153] Third, the second cross section acquisition unit 44 inputs the sample acquired as a candidate for the second cross section in the previous process to the second cross section acquisition classifier, and determines whether the input sample is a correct sample or an incorrect sample. The third process in step S804 is the same as the third process in step S304.

[0154] As in step S304, the second process and the third process are executed a predetermined number of times (for example, 10,000 times). If there are multiple samples determined to be correct, the second cross-section acquisition unit 44 outputs the average position and the average normal vector of each cross-section as the output of this step. If no sample determined to be correct is found, the second cross-section acquisition unit 44 displays a dialog box on the display unit 36 ​​to notify the user that no second cross-section was found.

[0155] Note that for a cross section in which the reconstruction error is smaller than a predetermined threshold and it is determined that the second cross section acquisition is not to be performed, the second cross section is fixed to the first cross section acquired in step S803, but this is not limited thereto. The second cross section acquisition unit 44 may not use the first cross section as the second cross section as is, but may impart a slight variation when cutting out a sample in step S805. That is, when cutting out a sample in step S805, the second cross section acquisition unit 44 may vary the first cross section within a range (e.g., ±3 degrees) smaller than the range (e.g., ±10 degrees) in which a cross section with a reconstruction error equal to or greater than a predetermined threshold can be taken. The second cross section acquisition unit 44 can set a different variation range for each cross section, thereby enabling more flexible estimation of the second cross section.

[0156] In steps S3024 and S805, the second cross-section acquisition unit 44 acquires the second cross-section using ERT, that is, by randomly extracting samples and determining whether they are correct or incorrect, but is not limited to this. The second cross-section acquisition unit 44 may acquire the second cross-section by a method of optimizing a predetermined evaluation function.

[0157] In the method for optimizing a predetermined evaluation function, the second cross section acquisition unit 44 applies principal component analysis (PCA) to the correct sample in step S3024 to obtain cross sections of the A, B, and C planes. We construct a model that captures the statistical trends of image features.

[0158] In step S804, the second cross-section acquisition unit 44 uses an evaluation function that uses two terms as costs: an error when the input cross-section image is re-expressed in each model, and a regularization term (angle difference from the first cross-section). For cross-sections of cases for which it is determined in step S804 that "second cross-section acquisition will not be performed," the second cross-section acquisition unit 44 increases the weight of the regularization term so as to minimize deviation from the first cross-section. The second cross-section acquisition unit 44 searches for parameters that minimize the cost value using a known optimization method such as the gradient method or the Levenburg-Marquardt method.

[0159] According to the third embodiment described above, whether or not to perform acquisition of the second cross sections is determined for each cross section, so that a group of second cross sections with higher accuracy can be acquired.

[0160] <Fourth embodiment> The image processing device 10 according to the fourth embodiment acquires (estimates) two or more predetermined second cross sections from a three-dimensional image, similarly to the first to third embodiments. In the first embodiment, when acquiring the second cross sections, in order to prevent the second cross sections from deviating too much from the angular relationship in which they are approximately orthogonal to each other, the second cross section acquisition unit 44 restricts the search range for a solution so that the deviation from the angular relationship between the cross sections falls within a predetermined range.

[0161] In contrast, in the fourth embodiment, in order to prevent the second cross sections from deviating too much from the orthogonal angular relationship, the image processing device 10 uses an evaluation function for acquiring the second cross sections to evaluate the degree of deviation from at least the orthogonal relationship between the cross sections. That is, the second cross section acquisition unit 44 acquires the first cross sections and the second cross sections by optimizing (minimizing or maximizing) the value of the evaluation function using an evaluation function including a term that quantifies the degree of deviation from the orthogonal relationship.

[0162] The configuration of the image processing device 10 according to the fourth embodiment is the same as that of the first embodiment shown in Fig. 2. The second cross-section acquisition process according to the fourth embodiment is the same as that of the first embodiment shown in Fig. 3 and the flowchart of Fig. 3. However, the construction process of each learning model shown in steps S3023 and S3024 and the acquisition process of each cross-section shown in steps S303 and S304 are partially different from those of the first embodiment. Each process will be explained below, clarifying the differences from the first embodiment.

[0163] (Step S3023: Construction of a learning model for obtaining the first cross section) In step S3023, the learning model generation unit 42 constructs a learning model using the learning data obtained in step S3022, in which the second cross-sectional information has been orthogonalized. The learning model generation unit 42 constructs a learning model that acquires the statistical tendency of the first cross-sectional image using principal component analysis (PCA), which is a known statistical process.

[0164] The specific procedure for constructing a learning model is described below. First, similar to step S3023 in the first embodiment, the learning model generation unit 42 generates a sample by cutting out a part of the 3D image from the learning case data using the second cross-sectional parameters (correct position) orthogonalized to the 3D image of the learning case data. As in the first embodiment, the learning model generation unit 42 also considers samples that have been slightly shifted in position, posture, or scale from the correct position as correct samples, and pads the learning data.

[0165] However, unlike step S3023 in the first embodiment, the learning model generation unit 42 does not generate non-correct samples. The learning model generation unit 42 generates learning samples by extracting correct samples from all learning case data. By performing principal component analysis on the obtained learning samples, a learning model that expresses the statistical tendency of the first cross-sectional image is constructed.

[0166] (Step S3024: Construction of a learning model for obtaining a second cross section) In step S3024, the learning model generation unit 42 constructs a learning model using the learning data (before being orthogonalized) obtained in step S3021. The learning model generation unit 42 constructs a learning model by principal component analysis (PCA) in the same manner as in step S3023 of the fourth embodiment.

[0167] The process of extracting sample images from the input image is the same as step S3024 in the first embodiment. However, the learning model generation unit 42 does not generate non-correct samples, but instead constructs a learning model using only correct samples, as in step S3023 in the fourth embodiment. The learning model generation unit 42 performs principal component analysis on the generated learning samples to construct a learning model that expresses the statistical trends of the second cross sections of each of the A, B, and C planes.

[0168] (Step S303: Calculation of the first cross section group) In step S303, the first cross-section acquisition unit 43 acquires a first group of cross sections using the input image acquired in step S301 and the learning model for first cross-section acquisition generated in step S3023. As in the first embodiment, the first cross-section acquisition unit 43 calculates a total of six parameters shown in equation (2).

[0169] The first cross-section acquisition unit 43 acquires the first cross-section by searching for parameters (six parameters shown in equation (2)) that minimize the cost function using the generated learning model. The cost function is the difference (reconstruction error) when a sample cut out from the input image is re-expressed using the model. In other words, the cost function is an index that quantifies "how likely the sample cut out from the input image is to be the first cross-section based on the statistical tendency of the learning model." The processing procedure for acquiring the first cross-section is shown below.

[0170] First, if image processing was applied when the learning model was constructed, the first cross-section acquisition unit 43 applies the same image processing to the input image. For example, if the learning model is constructed using differential images, the first cross-section acquisition unit 43 also performs differential processing on the input image.

[0171] Next, the first cross-section acquisition unit 43 determines the initial values ​​of the six parameters (hereinafter referred to as initial parameters) shown in equation (2). For example, the initial value of the position vector C is set to the center of the image, and the initial value of the parameter R indicating the orientation is set to a direction parallel to each axis of the input image. The first cross-section acquisition unit 43 searches for parameters that minimize the cost value using a known optimization method such as the gradient method or the Levenburg-Marquardt method. The parameters of the searched first cross-section are the position vector Cc and the parameter Rc.

[0172] The first cross-section acquisition unit 43 may, for example, calculate three position parameters, fix the calculated position parameters, and then calculate three angle (posture) parameters. Conversely, the first cross-section acquisition unit 43 may calculate three angle parameters, fix the calculated angle parameters, and then calculate three position parameters.

[0173] (Step S304: Obtaining the second cross section group) In step S304, the second cross-section acquisition unit 44 acquires each second cross-section using the input image acquired in step S301, the first cross-section parameters Cc and Rc calculated in step S303, and the second cross-section acquisition learning model generated in step S3024.

[0174] For example, the second cross-section acquisition unit 44 can acquire the second cross-sections by minimizing the cost function and calculating (correcting) the orientation of each second cross-section represented by a total of 12 parameters shown in equation (1) in the first embodiment. The cost function includes two types of terms: the reconstruction error described in step S303 of the fourth embodiment, and a constraint term that imposes a penalty when the positional relationship between the cross-sections deviates from the orthogonal relationship.

[0175] The reconstruction error is the same as that described in step S303 of the fourth embodiment. The constraint term can be the absolute value of the dot product of the normal vectors representing the cross sections. When the cross sections are orthogonal to each other, the dot product of the normal vectors is 0, so the degree of deviation from the orthogonal relationship can be quantified by taking the absolute value of the dot product. When there are three cross sections, namely, plane A, plane B, and plane C, the constraint term is expressed as the sum of the dot products of the three combinations of the plane A normal vector and plane B normal vector, the plane B normal vector and plane C normal vector, and the plane C normal vector and plane A normal vector. The processing procedure is shown below.

[0176] First, the second cross section acquisition unit 44 calculates the angle parameter Rc among the first cross section parameters by calculating the normal vector N A c, N B c, N C c. This process is the same as the process in step S3041 in the first embodiment.

[0177] Second, if image processing was applied when the learning model was constructed, the second cross-section acquisition unit 44 applies the same image processing to the input image. For example, if the learning model is constructed using differential images, the second cross-section acquisition unit 44 also performs differential processing on the input image.

[0178] Next, the second cross-section acquisition unit 44 determines initial parameters that serve as starting points for acquiring each of the A, B, and C planes. The second cross-section acquisition unit 44 fixes the position vector Cc and the normal vector N A c is the initial parameter of the A surface, and the normal vector N B c is the initial parameter of the B surface, and the normal vector N C Let c be the initial parameter of the C surface.

[0179] Third, the second cross-section acquisition unit 44 searches for the parameters (normal vectors) of each of the A, B, and C surfaces by minimizing a cost function. The cost function is defined as (reconstruction error of A surface + reconstruction error of B surface + reconstruction error of C surface) + constraint term. In the cost function, the reconstruction error term and constraint term are adjusted by a weight parameter that is set in advance. The second cross-section acquisition unit 44 searches for the parameters of each of the A, B, and C surfaces by known optimization methods such as the gradient method and the Levenburg-Marquardt method.

[0180] In step S304, a constraint term was introduced in which the cost increases as the cross sections deviate from the orthogonal relationship. However, if the angular relationship that the cross sections must satisfy is not orthogonal, the constraint term is defined according to the angular relationship that the cross sections must satisfy. For example, if surface A and surface B must satisfy an 80-degree angular relationship, the value of the constraint term can be the absolute value of the angle difference between surface A and surface B minus 80 degrees.

[0181] In this way, the image processing device 10 can calculate the parameters of each of the A plane, the B plane, and the C plane, and acquire the second cross-section group.

[0182] Although both the first and second cross-sectional groups were obtained using a learning model based on principal component analysis (PCA), either one may use a learning model generated by an ERT-based method. For example, the second cross-sectional group may be obtained using a PCA-based learning model, and the first cross-sectional group may be obtained using an ERT-based learning model. When the range of the first cross-sections is narrowed to a certain extent, the first cross-sections can be obtained more quickly by using ERT, which can build a learning model with fewer cases than PCA.

[0183] According to the fourth embodiment described above, in order to prevent the second cross sections from deviating too much from the orthogonal angular relationship, the evaluation function for calculating the group of second cross sections includes a constraint term for evaluating the degree of deviation from the orthogonal relationship. In acquiring the second cross sections, the cost (constraint term) increases as the cross sections deviate from the orthogonal relationship. Therefore, the image processing device 10 can acquire second cross sections that balance the statistical probability obtained from the learning model with the orthogonality of the cross sections.

[0184] The above-described embodiments merely exemplify the configuration of the present invention, and the present invention is not limited to the specific embodiments described above, and various combinations and modifications are possible within the scope of the technical concept thereof.

[0185] <Other embodiments> Furthermore, the disclosed technology can be embodied as, for example, a system, a device, a method, a program, or a recording medium (storage medium), etc. Specifically, the disclosed technology may be applied to a system made up of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to an apparatus made up of a single device.

[0186] The object of the present invention is also achieved as follows: A recording medium on which is recorded software program code (computer program) that realizes the functions of each of the above-described embodiments is supplied to a system or device. The recording medium is readable by a computer. The computer (or CPU or MPU) of the system or device reads and executes the program code stored on the recording medium. In this case, the program code itself read from the recording medium realizes the functions of each of the above-described embodiments. The recording medium on which the program code is recorded constitutes the present invention.

[0187] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0188] 10 Image processing device 32 ROM 33 RAM 37 Control Unit 41 Image acquisition unit 43 First cross-section acquisition section 44 Second cross-section acquisition section

Claims

1. an image acquisition unit that acquires an input image that is a three-dimensional image of an object; a cross-section acquisition unit that acquires a first cross-section group consisting of a plurality of first cross-sections that are set for observing the object in the input image and have a predetermined angular relationship with each other, and a second cross-section group consisting of a plurality of second cross-sections based on a constraint condition based on the predetermined angular relationship; An image processing device comprising:

2. The second cross section is a cross section obtained by varying the first cross section.

2. The image processing device according to claim 1, wherein:

3. The cross section acquisition unit a first learning model that receives an input of a partial three-dimensional image cut out from the input image and outputs whether the partial three-dimensional image is appropriate as a three-dimensional image for identifying the first group of cross sections, and acquires the first group of cross sections by inputting the partial three-dimensional image cut out from the input image into the first learning model; a second learning model for each cross section that receives a cross section image as an input and outputs whether the cross section image is appropriate as the second cross section, and acquires the second cross section group by inputting cross section images cut out from the input image based on the first cross section group into the second learning model; 3. The image processing device according to claim 1, wherein the image processing device is a computer.

4. When a plurality of candidates for a first group of cross sections determined to be appropriate by the first learning model are obtained, the cross section acquisition unit acquires the first group of cross sections by averaging the plurality of candidates for the first group of cross sections.

4. The image processing device according to claim 3.

5. When a plurality of second cross-section candidates determined to be appropriate are obtained in the second learning model for each cross-section, the cross-section acquisition unit acquires the second cross-section corresponding to each cross-section by averaging the plurality of second cross-section candidates.

5. The image processing device according to claim 3, wherein the image processing device is a computer.

6. The cross section acquisition unit inputs cross sections cut out within a range of a predetermined angle centered on a normal vector of each of the first cross sections of the first cross section group into the second learning model.

6. The image processing device according to claim 3, wherein the image processing device is a computer.

7. The cross section acquisition unit a first learning model that receives an input of a partial three-dimensional image cut out from the input image and outputs whether the partial three-dimensional image is appropriate as a three-dimensional image for identifying the plurality of first cross sections, and acquires the first cross section group by inputting the partial three-dimensional image cut out from the input image into the first learning model; a second learning model that receives an input of a partial three-dimensional image obtained by cutting out a part of the input image and outputs whether the partial three-dimensional image is appropriate as a three-dimensional image for identifying the second cross section, and acquires the second cross section group by inputting the partial three-dimensional image obtained by cutting out a part of the input image into the second learning model based on the first cross section group; 2. The image processing device according to claim 1, wherein:

8. The first learning model and the second learning model are generated using an image to which at least one of image processing including differentiation, noise reduction, and sharpening has been applied.

8. The image processing device according to claim 3, wherein the image processing device is a computer.

9. The cross section acquisition unit applying the image processing applied to the image used to generate the first learning model to the input image to obtain the first group of cross sections; The image processing applied to the image used to generate the second learning model is applied to the input image to obtain the second cross-section group.

9. The image processing device according to claim 8,

10. The cross-section acquisition unit acquires a second cross-section group consisting of the plurality of second cross-sections, using the plurality of first cross-sections as initial values ​​of the respective second cross-sections, under the constraint that the second cross-sections do not deviate too much from the predetermined angular relationship.

10. The image processing device according to claim 1, wherein the image processing device is a computer.

11. The cross-section acquisition unit acquires the second cross-section group so that deviation of an angular relationship between the second cross-sections included in the second cross-section group from the predetermined angular relationship falls within a predetermined angle range.

11. The image processing device according to claim 1,

12. The evaluation function used by the cross section acquisition unit to acquire the second cross section group evaluates the degree of deviation between the angular relationship of the second cross sections included in the second cross section group and the predetermined angular relationship.

12. The image processing device according to claim 1, wherein the image processing device is a computer.

13. a display control unit that displays, on a display unit, cross-sectional images obtained by cutting the input image at the second cross sections included in the second cross-section group, and information indicating a degree of deviation between an angular relationship of the second cross sections included in the second cross-section group and the predetermined angular relationship.

13. The image processing device according to claim 1, wherein the image processing device is a computer.

14. a determination unit that determines whether the first cross-section group satisfies a predetermined condition for being used as the second cross-section group, When the first group of cross sections satisfies the predetermined condition, the cross section acquisition unit sets the first group of cross sections as the second group of cross sections.

14. The image processing device according to claim 1,

15. a determination unit that determines whether each of the first cross sections included in the first cross section group satisfies a predetermined condition for being used as the second cross section, The cross-section acquisition unit acquires the second group of cross sections without changing the first cross sections that satisfy the predetermined condition.

15. The image processing device according to claim 1,

16. the cross-section acquisition unit sets a weight of a regularization term for the first cross-section that satisfies the predetermined condition to a weight greater than a regularization term for the first cross-section that does not satisfy the predetermined condition in an evaluation function used to acquire the second cross-section group, thereby preventing the second cross-section from deviating from the first cross-section that satisfies the predetermined condition.

16. The image processing device according to claim 15.

17. a display control unit that displays information about the second cross-section group acquired by the cross-section acquisition unit on a display unit.

17. The image processing device according to claim 1,

18. The display control unit displays one second cross section included in the second cross section group on the display unit, and displays a line representing another second cross section intersecting the one second cross section in a superimposed manner on the one second cross section.

18. The image processing device according to claim 17,

19. The display control unit displays one second cross section included in the second cross section group on the display unit, and displays information on the degree of deviation, which indicates how much another second cross section intersecting with the one second cross section deviates from an angle perpendicular to the one second cross section, on the one second cross section by superimposing the information on the degree of deviation.

19. The image processing device according to claim 17 or 18.

20. The display control unit displays the information on the degree of deviation by character information or a display mode of the color or thickness of the line representing the other second cross section.

20. The image processing device according to claim 19,

21. The cross-section acquisition unit acquires images input by a user operation as the first cross-section group.

21. The image processing device according to claim 1,

22. The image acquisition unit acquires, as the input image, an image designated by a user operation or an image automatically selected based on a predetermined criterion.

22. The image processing device according to claim 1,

23. a first generation unit that receives a three-dimensional image as an input, and generates a first learning model for acquiring a first cross section, and outputs whether a partial three-dimensional image obtained by cutting out a part of the three-dimensional image is appropriate as a three-dimensional image for identifying a first cross-section group including a plurality of first cross sections that are in a predetermined angular relationship with each other; a second generation unit that receives a cross-sectional image as an input and generates a second learning model for acquiring a second cross-section for each cross-section, and outputs whether the cross-sectional image is appropriate as a second cross-section for observing an object; 1. An image processing device comprising:

24. The first generation unit receives information on a three-dimensional image and a correct second cross section as input, generates correct samples and non-correct samples based on information on a correct position corrected so that the correct second cross section has the predetermined angular relationship, and generates the first learning model by learning the correct samples and the non-correct samples.

24. The image processing device according to claim 23.

25. The second generation unit receives a three-dimensional image and correct second cross-sectional information as input, generates correct samples and incorrect samples of two-dimensional images for each cross-section, and generates the second learning model for each cross-section by learning the correct samples and the incorrect samples.

25. The image processing device according to claim 23 or 24.

26. The computer An input image is obtained, which is a three-dimensional image of the object; A first group of cross sections is set for observing the object in the input image and is made up of a plurality of first cross sections that are in a predetermined angular relationship with each other, and a second group of cross sections is acquired based on a constraint condition based on the predetermined angular relationship. An image processing method comprising:

27. A program for causing a computer to execute the image processing method according to claim 26.

Images