Image processing apparatus, image processing method, and program

Abstract

To improve accuracy of region extraction related to a region of a subject from an image obtained by imaging the region.SOLUTION: An image processing apparatus according to the present disclosure includes an image acquisition unit configured to acquire a plurality of images in which a part of a subject is imaged, an initial region acquisition unit configured to acquire information about a plurality of initial regions corresponding to the part from a predetermined number of images among the plurality of images, and a detail region acquisition unit configured to acquire information about a detail region corresponding to the part in any of the plurality of images based on the information about the plurality of initial regions.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In diagnosis using 3D medical images, doctors calculate various diagnostic indices by identifying the positions of specific anatomical landmarks (feature points) in cross-sectional (reference cross-sectional) images used for observation and by extracting specific anatomical structures (regions). In recent years, advances in machine learning technologies such as deep learning have made it possible to identify anatomical landmarks and extract anatomical structures with high accuracy, provided that large amounts of training data with sufficient variation are available.

[0003] However, unlike general images, for which it is relatively easy to collect a large amount of training data, medical images pose a challenge in that it is difficult to easily collect a large amount of training data that includes variations of the target object. For this reason, a technology has been proposed that enables effective learning from a small amount of training data by increasing the training data of medical images (data augmentation).

[0004] For example, in Patent Document 1, reference cross-section parameters indicating the position of a reference cross-section in a three-dimensional image are varied, cross-sectional images are generated using the varied parameters, correct answer data on the cross-sectional images is calculated, and the correct answer data is added to the training data to augment the training data. Also, in Non-Patent Document 1, an estimator (learning model) trained with data for which the correct answer is known is used to make an estimation for data for which the correct answer is unknown, thereby creating pseudo-correct answer data to augment the training data. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2023-135836 [Non-Patent Document 1] Qizhe Xie, et al. “Self-training with Noisy Student improves ImageNet classification”, CVPR, 2020. Summary of the Invention [Problem to be solved by the invention]

[0006] However, the above-described conventional methods do not take into account the positional relationship between the imaging probe and the subject in the ultrasound image, and therefore may generate images that cannot be captured in actual ultrasound imaging. Also, it is difficult to effectively generate variations corresponding to the variations that can be captured in actual ultrasound imaging.

[0007] The technology of the present disclosure has been made in view of the above, and aims to improve the accuracy of estimating information about an object from an image of a subject. [Means for solving the problem]

[0008] The image processing device according to the present disclosure includes an image acquisition unit that acquires a reference cross-sectional image from each of a plurality of three-dimensional images including an object; a training data acquisition unit that acquires training data including the reference cross-sectional image and ground truth data that is information about the object in the reference cross-sectional image; a first learning unit that uses the training data to learn a first learning model that estimates information about the object from a cross-sectional image in the three-dimensional image to acquire a first trained model; and a first learning unit that, for each of the plurality of three-dimensional images, calculates a first trained model based on a relationship between an imaging plane of a three-dimensional imaging probe that images the object in the three-dimensional image and a reference cross-section of the reference cross-sectional image. The image processing device includes an image processing device having a pseudo image acquisition unit that acquires different cross sections as pseudo reference cross section images, a pseudo ground truth data acquisition unit that acquires pseudo ground truth data, which is information about the object in the pseudo reference cross section image, using the first trained model, and a second learning unit that acquires a second trained model by learning a second learning model that estimates information about the object from a cross section image in the three-dimensional image using pseudo teacher data including the pseudo reference cross section image and the pseudo ground truth data.

[0009] The image processing device according to the present disclosure also includes an image processing device characterized by having: a trained model acquisition unit that uses teacher data including a reference cross section image acquired from a 3D image including an object and ground truth data that is information about the object to train a first learning model that estimates information about the object from a cross section image in the 3D image to acquire a first trained model; acquires a pseudo reference cross section image that intersects with an imaging plane by the 3D imaging probe in the 3D image and differs from the reference cross section; acquires information about the object estimated using the pseudo reference cross section image and the first trained model as pseudo ground truth data; trains a second learning model using pseudo teacher data including the pseudo reference cross section image and the pseudo ground truth data to acquire a second trained model that estimates information about the object from a cross section image in the 3D image; an input image acquisition unit that acquires a reference cross section as an input image from a 3D image of the object captured using the 3D imaging probe; and an estimation unit that estimates information about the object using the input image and the second trained model.

[0010] The image processing device according to the present disclosure also includes an image processing device comprising: an image acquisition unit that acquires a reference cross section image from each of a plurality of 3D images including an object; a training data acquisition unit that acquires, for each of the reference cross section images, training data including the reference cross section image and ground truth data that is information about the object in the reference cross section image; a pseudo image acquisition unit that acquires, for each of the plurality of 3D images, a cross section different from the reference cross section as a pseudo reference cross section image based on a relationship between an imaging plane of a 3D imaging probe that images the object in the 3D image and a reference cross section of the reference cross section image; a pseudo training data acquisition unit that acquires pseudo training data including the pseudo reference cross section image and the ground truth data corresponding to the reference cross section image used for acquiring the pseudo reference cross section image by the pseudo image acquisition unit; and a learning unit that uses the pseudo training data to train a learning model that estimates information about the object from a cross section image in the 3D image.

[0011] The image processing method according to the present disclosure also includes an image acquiring step of acquiring a reference cross-sectional image from each of a plurality of three-dimensional images including an object; a training data acquiring step of acquiring training data including the reference cross-sectional images and ground truth data that is information about the object in the reference cross-sectional images; a first learning step of acquiring a first trained model by using the training data to learn a first model that estimates information about the object from cross-sectional images in the three-dimensional images; and an imaging step of acquiring an image of the object in each of the plurality of three-dimensional images using a three-dimensional imaging probe. and a second learning step of acquiring a second learned model by using pseudo teacher data including the pseudo reference cross section image and the pseudo ground truth data to learn a second learned model that estimates information about the object from a cross section image in the three-dimensional image.

[0012] The image processing method according to the present disclosure also includes: acquiring a first trained model by using teacher data including a reference cross-sectional image acquired from a three-dimensional image including an object and correct answer data that is information about the object, to learn a first trained model that estimates information about the object from a cross-sectional image in the three-dimensional image; and acquiring a first trained model by using teacher data including a reference cross-sectional image acquired from the three-dimensional image including an object and correct answer data that is information about the object. and acquiring pseudo reference cross section images that intersect with and are different from the reference cross section, acquiring information about the object estimated using the pseudo reference cross section images and the first trained model as pseudo-ground-truth data, and training a second trained model using pseudo teacher data including the pseudo reference cross section images and the pseudo-ground-truth data to acquire a second trained model that estimates information about the object from cross section images in the three-dimensional image; an input image acquisition step of acquiring a reference cross section as an input image from a three-dimensional image of the object captured using the three-dimensional imaging probe; and an estimation step of estimating information about the object using the input image and the second trained model.

[0013] The image processing method according to the present disclosure also includes an image processing method including: an image acquiring step of acquiring a reference cross section image from each of a plurality of three-dimensional images including an object; a teacher data acquiring step of acquiring, for each of the reference cross section images, teacher data including the reference cross section image and ground truth data that is information about the object in the reference cross section image; a pseudo image acquiring step of acquiring, for each of the plurality of three-dimensional images, a cross section different from the reference cross section as a pseudo reference cross section image based on a relationship between an imaging plane of a three-dimensional imaging probe that images the object in the three-dimensional image and a reference cross section of the reference cross section image; a pseudo teacher data acquiring step of acquiring pseudo teacher data including the pseudo reference cross section image and the ground truth data corresponding to the reference cross section image used to acquire the pseudo reference cross section image by the pseudo image acquiring step; and a learning step of using the pseudo teacher data to train a learning model that estimates information about the object from a cross section image in the three-dimensional image. [Effects of the Invention]

[0014] According to the technology of the present disclosure, it is possible to improve the accuracy of estimating information about an object from an image of a subject. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of an image processing apparatus according to a first embodiment. [Figure 2] 3 is a flowchart of processing executed by the image processing apparatus according to the first embodiment. [Figure 3] FIG. 10 is a diagram showing a schematic configuration of an image processing apparatus according to a second embodiment. [Figure 4] 10 is a flowchart of processing executed by an image processing apparatus according to a second embodiment. [Figure 5] FIG. 2 is a diagram schematically showing a reference cross section of a three-dimensional ultrasound image according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, exemplary embodiments of the image processing device of the present invention will be described in detail with reference to the accompanying drawings. However, the components described in the following embodiments are merely examples, and the technical scope of the present invention is determined by the claims and is not limited to the following individual embodiments.

[0017] Hereinafter, the technology of the present invention will be described using as an example a case in which the present invention is implemented using a convolutional neural network (CNN), which is one type of machine learning and one type of estimator based on deep learning.

[0018] First Embodiment The image processing device according to the first embodiment is a device that performs learning of a learning model that estimates a group of contour points of an object from a reference cross section image in a three-dimensional image of the object captured using a three-dimensional imaging probe. Here, the reference cross section is a two-dimensional cross section suitable for observing the object in the three-dimensional image. In this embodiment, a case is assumed in which a three-dimensional ultrasound image of the heart captured using an ultrasound probe is input to the learning model and the right ventricle is used as the object in a cardiac examination (cardiac ultrasound examination) using an ultrasound diagnostic device. Below, the right ventricle cut out from the three-dimensional ultrasound image of the heart is used as the object. This paper describes a method for training a learning model that uses a reference cross-sectional image of the ventricle as input to estimate the two-dimensional coordinate values ​​of the contour points that make up the contour of the right ventricle on the reference cross-section.

[0019] In this embodiment, the image input to the learning model is a three-dimensional image, but the image and correct answer data used in learning the teacher model and student model described below are two-dimensional reference cross-sectional images cut out from the three-dimensional image. That is, the teacher model and student model generated by the method of this embodiment are learning models that input two-dimensional reference cross-sectional images and output information about the object in the two-dimensional reference cross-sectional image. Furthermore, the teacher model is a first learning model that estimates information about the object from cross-sectional images in the three-dimensional image, and the first trained model is obtained by training the teacher model. Furthermore, the student model is a second learning model that estimates information about the object from cross-sectional images in the three-dimensional image, and the second trained model is obtained by training the student model.

[0020] The configuration and processing of the image processing device of this embodiment will be described below with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the schematic configuration of an image processing system (also referred to as a medical image processing system) including the image processing device of this embodiment. As shown in Fig. 1, the image processing system 1 includes an image processing device 10 and a database 22.

[0021] The image processing device 10 is communicably connected to a database 22 via a network 21. The network 21 includes, for example, a LAN (Local Area Network) or a WAN (Wide Area Network).

[0022] The database 22 holds and manages images and information used in processing by the image processing device 10 in this embodiment. The information managed by the database 22 includes three-dimensional images acquired by the image processing device 10, reference cross section parameters used by a reference cross section image acquisition unit 42 (described later), and supervised data used by a teacher dataset acquisition unit 43. Note that information on the three-dimensional images, reference cross section parameters, and supervised data may be stored in an internal memory (ROM 32 or storage unit 34) of the image processing device 10 instead of in the database 22. The image processing device 10 acquires the data held in the database 22 via the network 21.

[0023] The image processing device 10 includes a communication IF (Interface) 31 (communication unit), a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33, a storage unit 34, an operation unit 35, a display unit 36, and a control unit 40.

[0024] The communication IF 31 (communication unit) is configured with a LAN card or the like and realizes communication between the image processing device 10 and external devices such as the database 22. The ROM 32 is configured with non-volatile memory or the like and stores various programs and various data. The RAM 33 is configured with volatile memory or the like and is used as work memory for temporarily storing programs and data currently being executed. The storage unit 34 is configured with an HDD (Hard Disk Drive) or the like and stores various programs and various data. The operation unit 35 is configured with a keyboard, mouse, touch panel, etc. and is an input unit that accepts instructions from users such as doctors and medical technicians. The display unit 36 ​​is configured with a display or the like and displays various information generated by the processing of the image processing device 10 in this embodiment to the user.

[0025] The control unit 40 is configured from a CPU (Central Processing Unit) or a dedicated or general-purpose processor. The control unit 40 may be configured from a GPU (Graphic Processing Unit) or the like. Alternatively, the control unit 40 may be configured from a FPGA (Field-Programmable Gate Array), ASIC (Application Specific Integrated Circuit) or the like. The control unit 40 includes a three-dimensional image acquisition unit 41, a reference cross section image acquisition unit 42, a teacher data set acquisition unit 43, a teacher model learning unit 44, a pseudo reference cross section image acquisition unit 45, a pseudo ground truth data acquisition unit 46, a pseudo teacher data set acquisition unit 47, and a student model learning unit 48, which will be described later.

[0026] The three-dimensional image acquisition unit 41 acquires a plurality of three-dimensional images to be processed and input to the image processing device 10 from the database 22 or the storage unit 34. The three-dimensional images may be acquired directly from a modality such as an ultrasound diagnostic device, and in this case, the image processing device 10 may be implemented in the modality as part of its functions.

[0027] The reference cross-sectional image acquisition unit 42 acquires reference cross-sectional parameters from the database 22 for each of the three-dimensional images acquired by the three-dimensional image acquisition unit 41. The reference cross-sectional image acquisition unit 42 then uses the acquired reference cross-sectional parameters to extract and acquire two-dimensional reference cross-sectional images from the three-dimensional image corresponding to the reference cross-sectional parameters. Here, the reference cross-sectional image is a two-dimensional cross-section suitable for observing an object, and the reference cross-sectional parameters are parameters representing the position and orientation of the reference cross-sectional image in the three-dimensional image. The reference cross-sectional parameters indicating the position of the reference cross-sectional image are, for example, a total of six parameters consisting of three three-dimensional coordinate parameters representing the center position of the reference cross-sectional image and three normal vector parameters representing the orientation of the reference cross-sectional image. Here, the reference cross-sectional parameters may be input by a user via the operation unit 35, or may be based on any estimation results using the three-dimensional image. The reference cross-sectional parameters representing the position and orientation of the reference cross-sectional image in the three-dimensional image are not limited to the above configuration and may be expressed in other forms (for example, a form expressed by a center position and a quaternion, Euler angles, or a rotation matrix representing rotation).

[0028] The teacher dataset acquisition unit 43 is a teacher data acquisition unit that acquires, from the database 22, supervised data, which is information about objects in the reference cross section images acquired by the reference cross section image acquisition unit 42. The teacher dataset acquisition unit 43 acquires a teacher dataset consisting of multiple pieces of teacher data, with a pair of a reference cross section image and supervised data as one piece of teacher data. Here, the supervised data may be information about objects manually set by the user for the acquired reference cross section images, or may be information estimated by the image processing device 10. In this embodiment, as a specific example of supervised data, a case will be described in which two-dimensional coordinate values ​​of a contour point group constituting the contour of the right ventricle depicted on a reference cross section image of the right ventricle extracted from a three-dimensional ultrasound image of the heart are acquired.

[0029] The teacher model learning unit 44 uses the teacher data set acquired by the teacher data set acquisition unit 43 to learn (construct) a learning model (estimator) that estimates information about the object from the reference cross-sectional image.

[0030] The pseudo reference cross section image acquisition unit 45 is a pseudo image acquisition unit that cuts out and acquires pseudo reference cross section images using the 3D images acquired by the 3D image acquisition unit 41 and the reference cross section parameters acquired by the reference cross section image acquisition unit 42. Specifically, for each 3D image, the pseudo reference cross section image acquisition unit 45 changes the position and orientation of the reference cross section in the 3D image represented by the reference cross section parameters, using the imaging plane of the 3D imaging probe in the 3D image as a reference. Then, the pseudo reference cross section image acquisition unit 45 cuts out images of cross sections different from the reference cross sections from the 3D image using the positions and orientations of the reference cross sections after the change, and acquires the cut-out cross section images as pseudo reference cross section images. In this way, the pseudo reference cross section image acquisition unit 45 acquires images of cross sections whose positions or orientations are different from those of the reference cross sections as pseudo reference cross section images. Alternatively, the pseudo reference cross section image acquisition unit 45 changes the normal vector of the reference cross section parameters to acquire images of cross sections whose normal directions are different from those of the reference cross sections as pseudo reference cross section images.

[0031] The pseudo-correct data acquiring unit 46 estimates information about the object using the learning model (also referred to as a teacher model) learned by the teacher model learning unit 44 for each of the pseudo reference cross section images acquired by the pseudo reference cross section image acquiring unit 45. Then, the pseudo-correct data acquiring unit 46 acquires the estimated information about the object as pseudo-correct data.

[0032] The pseudo teacher data set acquisition unit 47 is a pseudo teacher data acquisition unit that acquires a pseudo teacher data set consisting of a plurality of pseudo teacher data. Here, the pseudo teacher data is a set of the pseudo reference cross section image acquired by the pseudo reference cross section image acquisition unit 45 and the pseudo supervised data acquired by the pseudo supervised data acquisition unit 46.

[0033] The student model learning unit 48 learns a learning model (also referred to as a student model) that estimates information about an object from a reference cross-sectional image, using the teacher dataset acquired by the teacher dataset acquisition unit 43 and the pseudo teacher dataset acquired by the pseudo teacher dataset acquisition unit 47. Note that the student model learning unit 48 may learn the student model using only the pseudo teacher dataset, without using the teacher dataset. The student model learning unit 48 is a learned model acquisition unit that acquires a second learned model that estimates information about an object from a cross-sectional image in a 3D image.

[0034] An example of processing by the image processing device 10 in FIG. 1 will be described in detail below with reference to the flowchart in FIG.

[0035] (Step S110: Acquiring a 3D image) In step S110, the three-dimensional image acquisition unit 41 acquires from the database 22 a plurality of three-dimensional images including the object (here, the right ventricle of the subject) specified by the user via the operation unit 35, and stores them in the storage unit 34. In this embodiment, the three-dimensional images acquired in step S110 are three-dimensional ultrasound images of the heart captured using a three-dimensional imaging probe in a cardiac ultrasound examination. Here, the control unit 40 may display the acquired three-dimensional ultrasound images of the heart on the display unit 36.

[0036] (Step S120: Acquiring a reference cross section image) In step S120, for each of the three-dimensional images acquired in step S110, the reference cross section image acquiring unit 42 acquires reference cross section parameters corresponding to the three-dimensional image from the database 22. Then, the reference cross section image acquiring unit 42 uses the acquired reference cross section parameters to cut out and acquire two-dimensional reference cross section images from each three-dimensional image.

[0037] In this embodiment, the reference cross-sectional image acquisition unit 42 acquires reference cross-sectional parameters of a reference cross-section for observing the right ventricle for each 3D ultrasound image of the heart. Then, the reference cross-sectional image acquisition unit 42 uses the acquired reference cross-sectional parameters to extract and acquire an image of the reference cross-section of the right ventricle from each 3D ultrasound image. In this embodiment, the reference cross-section is a cross-section (four-chamber cross-section) that allows simultaneous observation of the left ventricle, left atrium, right ventricle, and right atrium of the heart. In this embodiment, the cross-section in which the apex of the heart is visualized directly below the tip of the probe and in which the size of the right ventricle is at its maximum is defined as the reference cross-section of the right ventricle. In the following description, this reference cross-section will be referred to as plane A. The control unit 40 may display the acquired reference cross-sectional images on the display unit 36.

[0038] 5A and 5B show a schematic diagram of plane A set in the 3D ultrasound image of the heart acquired in step S110, and the cardiac chambers, left ventricle, left atrium, right ventricle, and right atrium, depicted in plane A. FIG. 5A shows a 3D ultrasound image 501 of the heart acquired in step S110, and plane A 502, which is a reference cross section acquired in step S120. FIG. 5A also shows an imaging plane (hereinafter referred to as probe plane) 503 of a 3D imaging probe (ultrasound probe) 504 in the 3D ultrasound image 501. FIG. 5B also shows a secondary image of plane A 502. 5B shows the original cross-sectional image. A left ventricle 505, a left atrium 506, a right ventricle 507, and a right atrium 508 depicted on plane A 502, respectively.

[0039] In this embodiment, the position of the three-dimensional imaging probe 504 in the three-dimensional ultrasound image 501 is identified based on image information (such as the angle of view and the contour of the visualized region) to be captured. Specifically, it can be identified from the imaging outline (a sector in the illustrated example) of an image (such as volume data or a B-mode image) including the reference cross section of the three-dimensional ultrasound image 501. The position of the three-dimensional imaging probe 504 may also be identified using DICOM information and position information (such as position designation information obtained from an examination order or information recorded by a radiologist) attached to the three-dimensional ultrasound image 501. Then, the probe surface 503 in the three-dimensional ultrasound image 501 can be identified based on the position of the three-dimensional imaging probe 504.

[0040] In this embodiment, plane A is used as the reference plane, but the reference plane may be another plane depending on the purpose of observing the object. For example, assume that the object is the right ventricle. In this case, the reference plane for the right ventricle may be a plane (two-chamber plane) in which the apex of the right ventricle is visualized directly below the tip of the probe, the right ventricle and right atrium are visualized in a positional relationship generally perpendicular to plane A, and the size of the right ventricle is at its maximum (hereinafter referred to as plane B).

[0041] (Step S130: Learning the teacher model) In step S130, the teacher dataset acquisition unit 43 acquires, for each reference cross-sectional image acquired in step S120, supervised data, which is information about the object in the reference cross-sectional image, from the database 22 and stores the data in the storage unit 34. As a result, the teacher dataset acquisition unit 43 acquires a teacher dataset consisting of a set of the reference cross-sectional image and the supervised data (teacher data). In this embodiment, the supervised data is data indicating the two-dimensional coordinate values ​​of a group of discretely arranged contour points indicating the contour of the right ventricle in the cross-sectional image of plane A. The teacher dataset acquisition unit 43 may also acquire the supervised data by other methods. For example, the image processing device 10 may sequentially acquire images captured from time to time, and the user may operate the operation unit 35 to sequentially create supervised data for each acquired image, and the teacher dataset acquisition unit 43 may acquire the supervised data created by the user.

[0042] Furthermore, the teacher model learning unit 44 learns a teacher model, which is a learning model that estimates information about the object (coordinate values ​​of the contour point cloud of the right ventricle) from the reference cross-sectional image using the teacher dataset acquired by the teacher dataset acquisition unit 43. Then, the teacher model learning unit 44 stores the learned model obtained by learning the teacher model in the database 22 or the storage unit 34. In this embodiment, a CNN is used to construct the learning model, and a known network such as VGG16 or Dense Net is used.

[0043] In this embodiment, the teacher model learning unit 44 learns a learning model that estimates the coordinate values ​​of the contour point group of the right ventricle region, but it may also learn a learning model that estimates the coordinate values ​​of the contours of other ventricles or atria. Furthermore, the teacher model learning unit 44 is not limited to learning models that estimate the coordinate values ​​of contour points, and may also learn a learning model that estimates the coordinate values ​​of landmark positions such as valves included in the cardiac region.

[0044] (Step S140: Acquiring pseudo reference cross section images) In step S140, the pseudo reference cross section image acquisition unit 45 extracts and acquires pseudo reference cross section images from each of the three-dimensional images acquired in step S110, using the reference cross section parameters acquired in step S120 and the imaging plane of the three-dimensional imaging probe. Here, the pseudo reference cross section is a cross section different from the original reference cross section, and is a cross section that satisfies the conditions of being spatially close to the reference cross section and sharing the imaging plane of the three-dimensional imaging probe.

[0045] Acquisition of a pseudo reference cross section image in this embodiment will be described with reference to FIGS. 5C and 5D. First, FIG. 5C shows the three-dimensional ultrasound image 501 and the A-plane 502 shown in FIG. 5A. The pseudo reference cross section image acquisition unit 45 calculates an intersection line 509 between the probe plane 503 and the A-plane 502 shown in FIG. 5C using the reference cross section parameters acquired in step S120. Here, the intersection line 509 is perpendicular to the irradiation direction of an ultrasound beam (not shown) emitted from the probe plane 503 by the three-dimensional imaging probe 504. Next, the pseudo reference cross section image acquisition unit 45 varies the reference cross section parameters of the A-plane so as to rotate the A-plane 502 by a certain angle around the calculated intersection line 509 as the rotation axis, thereby calculating pseudo reference cross section parameters. Here, the pseudo reference cross section parameters are parameters expressed in the same form as the reference cross section parameters acquired in step S120. Then, the pseudo reference cross section image acquiring unit 45 acquires, as a pseudo reference cross section image 510, a cross section image cut out from the three-dimensional ultrasonic image 501 of the heart using the calculated pseudo reference cross section parameters.

[0046] Specifically, the pseudo reference cross section image acquiring unit 45 acquires a plurality of pseudo reference cross section images by rotating the A plane 502 at a predetermined increment within a predetermined angle range around the intersection line 509 as a rotation axis. For example, the pseudo reference cross section image acquiring unit 45 acquires 20 images of cross sections obtained by rotating the A plane at 1° increments within a range of 10° around the rotation axis (10 images obtained by rotating in the positive direction of the rotation angle and 10 images obtained by rotating in the negative direction of the rotation angle).

[0047] The angle range for rotating the reference cross section is merely an example. For example, the angle range of the error (inter-examiner error) of the reference cross section parameters that occurs when multiple doctors or technicians set the reference cross sections in the image processing device 10 may be adopted as the above-mentioned angle range. The number of pseudo reference cross section images acquired by the pseudo reference cross section image acquisition unit 45 may be an appropriate number as the number of images used for learning the student model described below. In the above example, multiple cross sections are acquired by rotating the reference cross section at a constant rotation angle. However, this is not limited to this in the present embodiment. For example, a rotation angle within a predetermined angle range set using random numbers or the like may be used. In this case, the rotation angle may be generated using a uniform distribution or a normal distribution using random numbers or the like. As a result, the pseudo reference cross section image acquisition unit 45 acquires, as pseudo reference cross section images, images including cross section images of cross sections other than the cross section obtained by rotating the orientation of the reference cross section around the axis of the irradiation direction of the ultrasound beam irradiated from the three-dimensional imaging probe.

[0048] According to the method described above, the pseudo reference cross section image acquisition unit 45 can acquire the pseudo reference cross section image 510 depicting the right ventricle 511 as a cross section different from the A plane 502, as shown in FIG. 5D.

[0049] In cardiac ultrasound examinations, the position on the subject at which the 3D imaging probe 504 for capturing a 3D ultrasound image of the heart is placed is predetermined for each imaging target (e.g., the apex, the left parasternal border, the epigastric region, etc.). Therefore, for example, in a 3D ultrasound image captured to observe the right ventricle of the heart, the positional relationship between a reference cross section, which is a standard cross section for observing the right ventricle, and the imaging plane of the 3D imaging probe is determined to some extent. In this embodiment, through the above processing, the pseudo reference cross section image acquisition unit 45 acquires a pseudo reference cross section image by rotating the reference cross section image based on the position of the imaging plane of the 3D imaging probe. As a result, the image processing device 10 can acquire a pseudo reference cross section that can be captured by a procedure intended to depict the reference cross section (i.e., that can be input when estimating the contour point cloud of the right ventricle).

[0050] In this embodiment, a cross section obtained by rotating the A plane around the intersection line between the probe plane and the A plane as the rotation axis is used as the pseudo reference cross section image, but the method of rotating the A plane in this embodiment is not limited to this. For example, the pseudo reference cross section image acquisition unit 45 may acquire a cross section obtained by translating the A plane as the pseudo reference cross section image. For example, the pseudo reference cross section image acquisition unit 45 may acquire a cross section obtained by translating the A plane as the pseudo reference cross section image. The plane A can be translated while being kept positioned within the probe plane 503. In this case, the probe plane 503 is the imaging plane of the three-dimensional imaging probe 504 and also the position of the body surface of the subject. Therefore, by translating the plane A within the probe plane 503, the plane A can be translated within the movement range of the three-dimensional imaging probe 504 on the body surface of the subject without moving the three-dimensional imaging probe 504. This allows the pseudo reference cross section image acquisition unit 45 to acquire, as the pseudo reference cross section, a cross section that is more likely to be the cross section obtained by the image captured by the three-dimensional imaging probe 504.

[0051] Furthermore, the pseudo reference cross section image acquiring unit 45 may calculate the rotation center for rotating the A plane using a random number or the like so that the rotation center is set near the probe position on the A plane with a high frequency, and acquire a cross section rotated around the calculated position as the pseudo reference cross section. Specifically, the pseudo reference cross section image acquiring unit 45 determines the rotation center using a probability distribution that increases the probability that the rotation center for rotating the A plane is set near the probe position, and acquires a cross section rotated around the determined rotation center as the pseudo reference cross section image. As a result, the positional relationship between the 3D imaging probe 504 and the subject in the pseudo reference cross section image is limited to a relationship suitable for estimating the contour point cloud of the right ventricle, and a suitable pseudo reference cross section image can be obtained by estimation using a learning model.

[0052] (Step S150: Generate pseudo-correct data) In step S150, the pseudo-ground-truth data acquiring unit 46 estimates information about the object by applying the trained model obtained by training in step S130 to each of the pseudo reference cross-sectional images acquired in step S140. Then, the pseudo-ground-truth data acquiring unit 46 acquires the estimated information about the object as pseudo-ground-truth data. In this embodiment, the pseudo-ground-truth data acquiring unit 46 estimates two-dimensional coordinate values ​​of the contour point cloud of the right ventricle in the pseudo reference cross-sectional images by a trained model trained using a teacher model, and acquires the estimated coordinate values ​​as pseudo-ground-truth data.

[0053] (Step S160: Learning the student model) In step S160, the pseudo teacher dataset acquisition unit 47 creates pseudo teacher data, which is a combination of the pseudo reference cross-sectional images acquired in step S140 and the pseudo ground truth data generated in step S150. The pseudo teacher dataset acquisition unit 47 then acquires a pseudo teacher dataset consisting of multiple pieces of pseudo teacher data. The student model learning unit 48 then uses the teacher dataset and pseudo teacher dataset acquired in step S130 to learn a student model, which is a learning model that estimates information about the object from the reference cross-sectional images. The student model learning unit 48 then stores the learned model obtained by learning the student model in the database 22 or the storage unit 34.

[0054] In this embodiment, a pseudo teacher data set consisting of a set of pseudo reference cross-sectional images of the right ventricle and coordinate values ​​of a contour point cloud of the right ventricle, which is pseudo ground truth data, is used to train a student model, which is a learning model that estimates coordinate values ​​of a contour point cloud of the right ventricle from a cross-sectional image of the right ventricle. Note that the model structure of the student model may be the same as that of the teacher model, or a model with a structure different from that of the teacher model may be used.

[0055] In this embodiment, a case has been described in which a student model is learned using only a pseudo teacher dataset consisting of pseudo reference cross section images and pseudo ground truth data acquired by the processing of steps S140 and S150. However, the method of learning the student model in this embodiment is not limited to this. For example, the student model learning unit 48 may also learn the student model using data acquired by a known data augmentation method in learning the above-mentioned student model. Specifically, the pseudo reference cross section image acquisition unit 45 performs processing such as translation, rotation, scaling, linear conversion of pixel values, and noise addition on the reference cross section image by a method different from step S140. The pseudo reference cross section image acquisition unit 45 acquires a second pseudo reference cross section image different from the pseudo reference cross section of step S150. More specifically, the pseudo reference cross section image acquisition unit 45 performs translation, rotation, scaling, etc. within the plane of the reference cross section, and rotates the reference cross section around the axis of the irradiation direction of the ultrasound beam irradiated from the 3D imaging probe. Then, the pseudo reference cross section image acquisition unit 45 acquires pseudo-ground-truth data (second pseudo-ground-truth data) for the second pseudo reference cross section image by a known method different from step S150. Then, the pseudo teacher dataset acquisition unit 47 creates second pseudo teacher data which is a set of the second pseudo reference cross section image and the second pseudo ground-truth data, and acquires a second pseudo teacher dataset consisting of a plurality of second pseudo teacher data. As a result, the student model learning unit 48 learns a student model using the pseudo teacher dataset and the second pseudo teacher dataset.

[0056] Here, it is desirable to train the postnatal model so that the pseudo teacher data set obtained by the processes of steps S140 and S150 has a higher influence on the learning of the student model than the second pseudo teacher data set obtained by a known data augmentation method. Specifically, for example, a larger number of pseudo teacher data sets than the number of second pseudo teacher data sets obtained by a known data augmentation method are obtained by the processes of steps S140 and S150. Alternatively, the second pseudo teacher data obtained by a known data augmentation method and the pseudo teacher data obtained by the processes of steps S140 and S150 may be weighted differently in the learning of the student model (more specifically, the weight used in calculating the loss function). In this case, the pseudo teacher data obtained by the processes of steps S140 and S150 is weighted higher than the weight given to the second pseudo teacher data obtained by a known data augmentation method. This allows student model learning unit 48 to make the influence of the pseudo teacher dataset acquired by the processes of steps S140 and S150 dominant in the learning of the student model, while also including pseudo teacher datasets acquired by other methods. As a result, student model learning unit 48 can acquire a more robust learned model by learning the student model.

[0057] In this embodiment, an object estimation process may be further performed using a trained model obtained by training the above-described student model, with an unknown cross-sectional image as input. Specifically, the control unit 40 functions as an input image acquisition unit, cuts out an image of plane A as a reference cross section from a 3D ultrasound image of the heart captured by echocardiography, and acquires the cut-out image as an input image. The control unit 40 then functions as an estimation unit, inputs the acquired input image into a trained model obtained by training the student model using the above-described method, and estimates the coordinate values ​​of the contour point cloud of the right ventricle. The control unit 40 may also display the plane A image, which is the input image, on the display unit 36 ​​and superimpose the coordinate values ​​of the estimated contour point cloud on the plane A image.

[0058] As described above, the processing of the image processing device 10 according to this embodiment enables training of a learning model capable of accurately estimating the two-dimensional coordinate values ​​of the contour point cloud of the right ventricle, which is an object, from a reference cross-sectional image in a 3D ultrasound image of the heart. As a result, the image processing device 10 can acquire a cross-section near the reference cross-section from the 3D ultrasound image of the heart as a pseudo reference cross-section based on the imaging plane of the 3D imaging probe. Then, cross-sectional images that appropriately reflect changes in the position of the 3D imaging probe can be augmented as data used for training the learning model. Furthermore, the image processing device 10 creates pseudo-ground-truth data for the pseudo reference cross-sectional image using a trained model obtained by training a teacher model with a teacher dataset. Thus, by acquiring the pseudo teacher dataset and training a student model for cross-sectional images other than the reference cross-section for which no ground-truth data is actually available, the image processing device 10 is expected to acquire a trained model with higher estimation accuracy of the contour point cloud of the right ventricle.

[0059] The following describes modifications of the above embodiment. In the following description, the same configurations and processes as those of the image processing system 1 of the above embodiment will be referred to as follows: The same reference numerals are used and detailed explanations are omitted.

[0060] (Variation 1-1) In the first embodiment, it is assumed that the processing target is a three-dimensional ultrasound image captured in a cardiac ultrasound examination. However, as in this modified example, the above embodiment can also be applied when using images captured with a three-dimensional imaging probe in an ultrasound examination other than that of the heart.

[0061] For example, assume that the image processing device 10 acquires a reference cross section image of the head of the fetus using a 3D ultrasound image of the fetus as an input image. Specifically, the reference cross section image acquisition unit 42 extracts and acquires a reference cross section image of the head from the reference cross section parameters of the head of the fetus in the 3D ultrasound image obtained by imaging the fetus with a 3D imaging probe. Then, the teacher model learning unit 44 learns a teacher model using the acquired reference cross section image as teacher data and the contour point cloud of the head region in the reference cross section image as ground truth data.

[0062] Next, the pseudo reference cross section image acquisition unit 45 acquires a cross section obtained by rotating the head reference cross section around the intersection line between the probe plane and the head reference cross section in the 3D ultrasound image of the fetus as a rotation axis, as a pseudo reference cross section image of the head. Then, the pseudo ground truth data acquisition unit 46 inputs the pseudo reference cross section image of the head into a trained model obtained by training the teacher model to estimate coordinate values ​​of the contour point cloud of the head region, and acquires the estimated coordinate values ​​of the contour point cloud as pseudo ground truth data. The pseudo teacher dataset acquisition unit 47 creates a pseudo teacher dataset using a pair of the pseudo reference cross section image and the pseudo ground truth data. Then, the student model training unit 48 trains a student model using the pseudo teacher dataset. As a result, the image processing device 10 of this modified example can augment the training data of the training model used to estimate the coordinate values ​​of the contour point cloud of the head region in the reference cross section image of the fetus's head.

[0063] In addition, in the image processing device 10 of this modified example, parts of the subject's heart other than the right ventricle, such as the left ventricle, left atrium, aortic valve, mitral valve, etc., can be used as the target, and the teacher model and student model can be trained using the same processing as above.

[0064] (Variation 1-2) In the first embodiment, it is assumed that an estimator based on deep learning, such as a CNN, is used as an estimator for realizing the teacher model and the student model. However, as in this modification, an estimator based on deep learning other than a CNN, such as a vision transformer, may also be used. In this modification, too, the teacher model and the student model can be learned in the image processing device 10 by processing similar to that in the first embodiment.

[0065] Alternatively, an estimator based on a known machine learning method other than deep learning, such as regression using Random Forest or Adaboost, may be used in the image processing device 10. Even in this case, the teacher model and the student model can be learned in the image processing device 10 by the same processing as in the first embodiment.

[0066] (Variation 1-3) In the first embodiment, it is assumed that a learning model is trained to input a reference cross section image of the right ventricle in a 3D ultrasound image captured by echocardiography and output coordinate values ​​of a contour point cloud of the right ventricle. However, the above embodiment can also be applied to a case where a learning model is trained to estimate information other than the coordinate values ​​of the contour points of an object in a reference cross section image, as in this modified example.

[0067] In this modification, for example, a case is assumed in which a region related to an object is estimated from a reference cross section image. Specifically, a case can be given in which a reference cross section image of the right ventricle in a 3D ultrasound image of the heart is input and the region of the right ventricle in the image is extracted. The processing procedure in this case will be explained. First, a teacher model is trained using a reference cross section image of the right ventricle extracted from a 3D ultrasound image of the heart using reference cross section parameters and information on the right ventricle region in the ground truth data as teacher data. Here, the information on the right ventricle region can be provided in the form of a mask image, for example. Next, a pseudo reference cross section image is extracted and acquired from the 3D ultrasound image using a method similar to that of the first embodiment. Then, the right ventricle region in the pseudo reference cross section image is estimated using the teacher model. Pseudo teacher data is created using the estimated region as pseudo ground truth data, and a student model is trained using the pseudo teacher dataset.

[0068] The mask image representing the right ventricle region, which is the correct data or pseudo-correct data, can be expressed as a binary value for each pixel (e.g., 0 for regions other than the right ventricle and 1 for the right ventricle). Furthermore, the pseudo-correct data may be an image in which the pixel values ​​are likelihood values ​​between 0 and 1 estimated by the teacher model, rather than the binary mask image described above. Unlike correct data accurately annotated by a doctor or technician, the pseudo-correct data is obtained by estimation using the teacher model. Therefore, the estimated region may include regions that are inappropriate for the right ventricle (falsely detected regions or overlooked regions). By using an image in which the pixel values ​​are likelihood values ​​estimated by the teacher model as pseudo-correct data, it is possible to reduce the impact on learning of low-likelihood regions that may be inappropriate for the right ventricle. Thus, this modification allows the above-described embodiment to be applied even when extracting a region of interest on a reference cross-sectional image.

[0069] (Variation 1-4) In the first embodiment, it is assumed that a learning model is trained to estimate coordinate values ​​of a contour point cloud representing the contour shape of the right ventricle using a reference cross section image of the right ventricle in an ultrasound cardiac image taken by an ultrasound cardiac examination as an input. However, as in this modification, the above embodiment can also be applied to training a learning model that estimates features other than the contour of an object in a reference cross section image.

[0070] In this modification, for example, a case is assumed in which the type of cross section of a reference cross section image extracted from a 3D ultrasound image of the heart is estimated (identified). Here, estimating (identifying) the type of cross section means estimating the type of cross section of a reference cross section image of the heart, which is an input image to a learning model. Examples of the cross section types of the reference cross section image include an apical four-chamber view (hereinafter referred to as a four-chamber view), an apical two-chamber view (hereinafter referred to as a two-chamber view), an apical three-chamber view (hereinafter referred to as a three-chamber view), a transsternal short-axis view (hereinafter referred to as a short-axis view), and a transsternal long-axis view (hereinafter referred to as a long-axis view). The apical four-chamber view is a cross section depicting the four chambers of the left ventricle, left atrium, right ventricle, and right atrium. The apical two-chamber view is a cross section depicting the two chambers of the left ventricle and left atrium. The apical three-chamber view is a cross section depicting the left ventricle, left atrium, and right ventricle. The transsternal short-axis view is a cross section perpendicular to the long axis connecting the left ventricular outflow tract and the apex of the heart. The transsternal long-axis view is a cross section along the long axis connecting the left ventricular outflow tract and the apex of the heart.

[0071] In this modification, it is assumed that the cross section type of the reference cross section image is estimated to be a four-chamber view, a two-chamber view, a three-chamber view, a short-axis view, or a long-axis view. In estimating the cross section type, a likelihood expressed from 0 to 1 is calculated for each cross section type, and the cross section type with the highest calculated likelihood is determined as the cross section type of the reference cross section image. As in the above embodiment, the teacher model learning unit 44 learns a teacher model using the reference cross section image acquired from a 3D ultrasound image of the heart using the reference cross section parameters and information indicating the cross section type of the cross section as teacher data.

[0072] Then, the pseudo reference cross section image acquiring unit 45 acquires pseudo reference cross section images by the same method as in the first embodiment. The pseudo teacher dataset acquiring unit 47 inputs the pseudo reference cross section images into the teacher model to estimate the cross section type, and acquires the estimated cross section type as pseudo supervised data. Then, the pseudo teacher dataset acquiring unit 47 creates pseudo teacher data consisting of a pair of the pseudo reference cross section images and the pseudo supervised data, and acquires a pseudo teacher dataset using the plurality of pseudo teacher data. The student model learning unit 48 learns the student model using the pseudo teacher dataset. do.

[0073] The cross-section type in the correct answer data is expressed as a binary value (for example, if the cross-section type is a four-chamber view, the output value corresponding to the four-chamber view is 1, and the output values ​​corresponding to other cross-section types are 0). However, the cross-section type in the pseudo-correct answer data is not limited to a binary value, and a likelihood value expressed as 0 to 1 estimated for each cross-section type may be used. The correct answer data differs from correct answer data accurately annotated by a doctor or technician, and is data based on likelihoods obtained by estimation using a teacher model. Therefore, there is a possibility that the correct answer data may represent an inappropriate cross-section type. In this modified example, the likelihood of the cross-section type estimated by the teacher model is further used as pseudo-correct answer data to train the student model, thereby reducing the impact of cross-section types with low likelihoods on the learning of the learning model.

[0074] (Variation 1-5) In the first embodiment, it is assumed that a student model is trained to estimate coordinate values ​​of a contour point cloud of the right ventricle using a reference cross section image of the right ventricle acquired from a 3D ultrasound image of the heart as an input. However, as in this modification, it is also possible to estimate information about an object using an unknown reference cross section image in which it is unknown whether information about the object exists or not, using a student model trained by a method similar to that of the first embodiment.

[0075] In this modification, information about the object is estimated by inputting an unknown reference cross-sectional image into a trained model obtained by training a student model using the same method as in Embodiment 1. As an example, in the image processing device 10, a reference cross-sectional image of the right ventricle acquired from a 3D ultrasound image of the heart is input into the student model to estimate the coordinate values ​​of the contour point cloud of the right ventricle.

[0076] In this modification, the student model learning unit 48 learns the student model using the method described in the first embodiment to acquire a learned model. The reference cross-sectional image acquisition unit 42 acquires a reference cross-sectional image of the right ventricle from a 3D ultrasound image in which the coordinate values ​​of the right ventricle contour point cloud are unknown. Here, the reference cross-sectional image can be identified by the user operating the operation unit 35 to specify the reference cross-sectional image of the right ventricle from the 3D ultrasound image of the heart, or by the control unit 40 estimating the reference cross-sectional image using a learned model that estimates reference cross-sectional parameters of the right ventricle from the 3D ultrasound image. The control unit 40 can then input the acquired reference cross-sectional image into the learned model obtained by learning the student model, and estimate the coordinate values ​​of the right ventricle contour point cloud.

[0077] In addition, the unknown reference cross section image in this modification does not necessarily have to be an image obtained by cutting out from a three-dimensional ultrasound image of the heart, but may be a two-dimensional ultrasound image obtained by the user imaging a reference cross section of the subject.

[0078] Furthermore, the process of learning a student model by the method described in the first embodiment and the estimation process using an unknown reference cross-sectional image do not necessarily have to be performed by the same device, but may be performed by different devices. In this case, for example, a trained model obtained by training a student model by the above process using a device other than the image processing device 10 is recorded in an arbitrary recording device. This allows the image processing device 10 to acquire the trained model recorded in the recording device and use the acquired trained model to execute the process of estimating information about the object in the unknown reference cross-sectional image by the above process.

[0079] (Variation 1-6) In the first embodiment, it is assumed that a learning model is obtained by executing the processes from step S110 to step S160. However, as in this modified example, for example, in the image processing device 10, after learning a student model by the processes from step S110 to step S160, the processes from step S140 onwards may be executed again. In this case, The learning model used to generate pseudo-supervised data in step S150 is not the teacher model obtained in step S130, but the student model already obtained in step S160.

[0080] As a result, in the image processing device 10 of this modified example, when step S160 is executed again, the student model used in the previous step S160 is trained using the generated pseudo-ground-truth data, thereby updating the trained model to one with higher estimation accuracy. That is, in the image processing device 10, in addition to the processing described in the first embodiment, the trained model based on the student model can be updated by repeatedly executing the processing from step S140 to step S160. As a result, it is expected that pseudo-ground-truth data with higher accuracy in information about the object can be generated, and a trained model that improves the estimation accuracy of information about the object can be obtained.

[0081] (Variation 1-7) In the image processing device 10 according to this modification, for example, after acquiring reference cross section images in steps S110 and S120, pseudo reference cross section images are acquired by processing in step S140. Then, the image processing device 10 may execute step S130 to perform training of the teacher model using the acquired reference cross section images and pseudo reference cross section images.

[0082] Specifically, assume that the image processing device 10 estimates (identifies) the cross-section type of a reference cross-sectional image extracted from a 3D ultrasound image of the heart. In this case, the teacher dataset acquisition unit 43 uses the same data as the correct answer data of the pseudo reference cross-sectional image as the correct answer data of the reference cross-sectional image. This allows the pseudo teacher dataset acquisition unit 47 to create a pseudo teacher dataset using a pair of the pseudo reference cross-sectional image and the correct answer data as teacher data. Then, in step S130, the teacher model learning unit 44 uses the created teacher dataset to learn the teacher model, thereby augmenting the learning data of the teacher model in the first embodiment in this example.

[0083] According to the image processing device 10 of this modified example, it is possible to increase the learning data used to learn the teacher model without creating pseudo-correct data using the teacher model, thereby improving the estimation accuracy of the teacher model and student model that estimate information about the target object.

[0084] Second Embodiment Next, an image processing system according to a second embodiment will be described. In the following description, the same components and processes as those in the first embodiment will be denoted by the same reference numerals, and detailed description thereof will be omitted.

[0085] Fig. 3 shows a schematic configuration of an image processing system 2 of this embodiment. As shown in Fig. 3, the image processing system 2 has an image processing device 20 and a database 22. The image processing device 20 of this embodiment differs from the image processing device 10 of the first embodiment shown in Fig. 1 in that it has a pseudo teacher dataset adjustment unit 51. In the following explanation, the processing performed by each unit of the image processing device 20 will be explained, focusing on differences from the processing content in the first embodiment.

[0086] As in the first embodiment, the image processing device 20 according to the second embodiment learns a learning model that estimates coordinate values ​​of a group of contour points of an object from a reference cross-sectional image in a three-dimensional image obtained by imaging the object using a three-dimensional imaging probe. The image processing device 10 according to the first embodiment does not perform any special selection process on the data used to learn the learning model, and instead learns a student model using, for example, a pseudo teacher data set made up of generated pseudo teacher data. On the other hand, the image processing device 20 according to this embodiment uses pseudo supervised data estimated using a trained model obtained by learning the teacher model as supervised data to be used to learn the student model. The image processing device 20 then acquires, as pseudo teacher data, a set of pseudo supervised data that has been determined to be suitable as supervised data to be used for learning the student model and pseudo reference cross-sectional images corresponding to the pseudo supervised data. The image processing device 20 also generates a pseudo teacher data set using the acquired plurality of pseudo teacher data.

[0087] In this embodiment, as in the first embodiment, a reference cross-section image of the right ventricle extracted from a 3D ultrasound image of the heart is used as an input image, and a learning model is assumed to learn the 2D coordinate values ​​of the contour point group that constitutes the contour of the right ventricle in the reference cross-section.

[0088] In this embodiment, the pseudo teacher dataset adjustment unit 51 determines whether the pseudo teacher data included in the pseudo teacher dataset acquired by the pseudo teacher dataset acquisition unit 47 is appropriate as data to be used for training a student model. The pseudo teacher dataset adjustment unit 51 then excludes from the pseudo teacher dataset any pseudo teacher data that it determines is not appropriate as data to be used for training a student model. In other words, the pseudo teacher dataset adjustment unit 51 adjusts the pseudo teacher dataset acquired by the pseudo teacher dataset acquisition unit 47 so that the pseudo teacher dataset is composed of only data that is appropriate as pseudo teacher data to be used for training a student model.

[0089] Next, an example of processing executed by the image processing device 20 will be described in detail using the flowchart in Fig. 4. As shown in Fig. 4, processing of step S210 for adjusting a pseudo teacher dataset is added to the processing executed by the image processing device 10 in the first embodiment shown in Fig. 2. Below, differences from the processing content in the first embodiment will be described in detail for each processing step executed by the image processing device 20.

[0090] (Step S210: Adjusting the pseudo teacher dataset) In step S210, the pseudo teacher dataset adjustment unit 51 determines whether the pseudo teacher data generated in step S150 is appropriate data based on the features of the reference cross section images acquired in step S120 and the supervised data generated in step S130. The pseudo teacher dataset adjustment unit 51 is a determination unit that determines, for each piece of pseudo teacher data, whether the pseudo teacher data is data from which information about the object in the pseudo reference cross section images has been appropriately estimated. If the pseudo teacher dataset adjustment unit 51 determines that the pseudo teacher data is inappropriate data, it excludes the pseudo teacher data from the pseudo teacher dataset. As a result, the excluded pseudo teacher data is not used in learning the student model in step S160.

[0091] In this embodiment, in step S210, for each coordinate value of the right ventricle contour point group estimated using the pseudo reference cross section image, the pseudo teacher dataset adjustment unit 51 determines whether the coordinate value indicates an appropriate position based on pixel values ​​surrounding the coordinate value. For example, the pseudo teacher dataset adjustment unit 51 calculates the edge strength of the position indicated by the coordinate value of the estimated contour point from pixel values ​​surrounding the position. Then, if the calculated edge strength is lower than a predetermined threshold, the pseudo teacher dataset adjustment unit 51 determines that the pseudo teacher data used to estimate the coordinate value is inappropriate data. In this case, the pseudo teacher dataset adjustment unit 51 can calculate the edge strength based on the spatial gradient of pixel values.

[0092] More specifically, the pseudo teacher dataset adjustment unit 51 calculates a contour line, i.e., a line connecting adjacent contour points, from the coordinate values ​​of the estimated contour point group. Then, the pseudo teacher dataset adjustment unit 51 calculates edge strength based on the spatial gradient of pixel values ​​in a direction perpendicular to the contour line. In addition, for example, the pseudo teacher dataset adjustment unit 51 calculates edge strength at multiple positions in a direction perpendicular to the contour line, and determines whether or not a position where the edge strength is equal to or greater than a predetermined value is located a predetermined distance away from the contour line. Then, the pseudo teacher dataset adjustment unit 51 calculates edge strength at multiple positions in a direction perpendicular to the contour line, and determines whether or not a position where the edge strength is equal to or greater than a predetermined value is located a predetermined distance away from the contour line. When it is determined that the position where the edge strength is equal to or greater than a predetermined value is a predetermined distance away from the contour line, the adjusting unit 51 determines that the pseudo-teacher data used to calculate the contour line is inappropriate data.

[0093] Alternatively, the pseudo teacher dataset adjustment unit 51 may determine the pseudo teacher data based on the relative positions of contour points, such as the curvature of a contour line calculated using the coordinate values ​​of the estimated contour point group. Specifically, the pseudo teacher dataset adjustment unit 51 calculates the curvature for each contour point based on the relative positions of the contour points on both sides. The pseudo teacher dataset adjustment unit 51 then determines that the pseudo teacher data used to calculate the curvature is appropriate if the calculated curvature does not deviate from the curvature distribution of the correct answer data and is within a certain range. Furthermore, the pseudo teacher dataset adjustment unit 51 determines that the pseudo teacher data used to calculate the curvature is inappropriate if the calculated curvature is outside the certain range of the curvature distribution of the correct answer data.

[0094] Alternatively, a user such as a doctor or technician can visually check the correct data to determine whether the coordinate values ​​of the estimated right ventricle contour point cloud indicate an appropriate position. For example, the control unit 40 displays pseudo teacher data on the display unit 36, and the user operates the operation unit 35 to input a judgment result (appropriate or inappropriate) for the pseudo teacher data displayed on the display unit 36. If the input user's judgment result indicates that the pseudo teacher data is "appropriate," the pseudo teacher dataset adjustment unit 51 includes the pseudo teacher data in the pseudo teacher dataset. On the other hand, if the input user's judgment result indicates that the pseudo teacher data is "inappropriate," the pseudo teacher dataset adjustment unit 51 excludes the pseudo teacher data from the pseudo teacher dataset. This allows the pseudo teacher dataset adjustment unit 51 to adjust the pseudo teacher dataset based on the user's visual confirmation.

[0095] Alternatively, whether the coordinate values ​​of the estimated right ventricle contour point cloud indicate an appropriate position can be determined using a classifier obtained by machine learning. Specifically, a classifier is trained in advance, which inputs the coordinate values ​​of contour points and estimates, as a likelihood, whether the input contour points are appropriate for the contour. The pseudo-teacher dataset adjustment unit 51 then inputs pseudo-teacher data to the trained classifier, and can determine that the coordinate values ​​of the contour point cloud made up of contour points indicate an appropriate position if the positions of the contour points of the object indicated by the pseudo-ground-truth data are equal to or greater than a predetermined likelihood.

[0096] Alternatively, the determination of whether the coordinate values ​​of the estimated right ventricle contour point cloud indicate an appropriate position may be made based on the supervised data acquired in step S130. For example, the pseudo teacher dataset adjustment unit 51 can determine whether the coordinate values ​​of the contour point cloud of the pseudo supervised data indicate an appropriate position by using the coordinate values ​​of the right ventricle contour point cloud of the reference cross section image in the 3D ultrasound image from which the pseudo reference cross section image is acquired.

[0097] For example, the pseudo teacher dataset adjustment unit 51 determines whether the pixel value at the position indicated by the coordinate values ​​of the contour point cloud of the pseudo ground truth data in the pseudo reference cross section image is within a certain range from the distribution of pixel values ​​around the position indicated by the coordinate values ​​of the contour point cloud of the right ventricle in the reference cross section image.The pseudo teacher dataset adjustment unit 51 then determines that the coordinate values ​​indicate an appropriate position when the pixel value at the position indicated by the coordinate values ​​of the contour point cloud of the pseudo ground truth data is within a certain range from the distribution of the pixel values.The pseudo teacher dataset adjustment unit 51 then includes the pseudo teacher data used to estimate the coordinate values ​​determined to indicate an appropriate position in the pseudo teacher dataset.Furthermore, the pseudo teacher dataset adjustment unit 51 excludes the pseudo teacher data used to estimate the coordinate values ​​determined not to indicate an appropriate position from the pseudo teacher dataset.

[0098] In this embodiment, as in the first embodiment, an object estimation process may be further performed using a trained model obtained by training the above-described student model, with an unknown cross-sectional image as input. Specifically, the control unit 40 cuts out an image of plane A as a reference cross section from a 3D ultrasound image of the heart captured by echocardiography, and acquires it as an input image. Next, in step S160, the acquired input image is input to a trained model obtained by training the student model, thereby estimating the coordinate values ​​of the contour point cloud of the right ventricle. Furthermore, the control unit 40 may display the plane A image, which is the input image, on the display unit 36, and superimpose the coordinate values ​​of the estimated contour point cloud on the plane A image.

[0099] By the processing described above, the image processing device 20 according to this embodiment can learn a student model that can more accurately estimate the coordinate values ​​of the contour point cloud of the right ventricle by learning using only pseudo-teacher data that is appropriate for learning.

[0100] <Other embodiments> Furthermore, the disclosed technology can be embodied as, for example, a system, a device, a method, a program, or a recording medium (storage medium), etc. Specifically, it may be applied to a system consisting of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or it may be applied to an apparatus consisting of a single device.

[0101] Needless to say, the object of the present invention can be achieved by the following: Namely, a recording medium (or storage medium) on which software program code (computer program) that realizes the functions of the above-described embodiments is recorded is supplied to a system or device. Needless to say, such a recording medium is a computer-readable recording medium. Then, a computer (or CPU or MPU) of the system or device reads and executes the program code stored on the recording medium. In this case, the recording medium on which the program code read from the recording medium is recorded constitutes the present invention.

[0102] In the above embodiment and modified examples, the various controls described as being performed by the control unit 40 may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0103] The above processor is a processor in a broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPs, U (Micro Processing Unit), DSP (Digital Signal Processor), etc. The processor is, for example, a GPU (Graphics Processing Unit). The dedicated processor is, for example, an ASIC (Application Specific Integrated Circuit), LD (Programmable Logic Device) and so on. Programmable logic devices are, for example, Examples include FPGA (Field Programmable Gate Array) and CPLD (Complex Programmable Logic Device).

[0104] Furthermore, although the embodiments and modifications of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, the above-described embodiments and modifications merely represent one embodiment of the present invention, and the above-described embodiments and modifications can be combined as appropriate.

[0105] The present invention provides a program for realizing one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and executes the program on a computer of the system or device. The present invention can be realized by a process in which one or more processors in the system read and execute a program, or by a circuit that implements one or more functions.

[0106] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an image acquisition unit that acquires a reference cross-sectional image from each of a plurality of three-dimensional images including the object; a training data acquisition unit that acquires training data including the reference cross-sectional image and correct answer data that is information about the object in the reference cross-sectional image; a first learning unit that uses the training data to learn a first learning model that estimates information about the object from a cross-sectional image in the three-dimensional image, and acquires a first learned model; a pseudo image acquisition unit that acquires, for each of the plurality of three-dimensional images, a cross section different from the reference cross section as a pseudo reference cross section image based on a relationship between an imaging plane of a three-dimensional imaging probe that images the object in the three-dimensional image and a reference cross section of the reference cross section image; a pseudo-ground-truth data acquisition unit that acquires pseudo-ground-truth data, which is information about the object in the pseudo reference cross section image, using the first trained model; a second learning unit that uses pseudo teacher data including the pseudo reference cross-sectional images and the pseudo ground truth data to learn a second learning model that estimates information about the object from cross-sectional images in the three-dimensional image, thereby acquiring a second learned model; 1. An image processing device comprising: (Configuration 2) the teacher data acquisition unit acquires, as the information about the object, information about the shape of a predetermined portion of the object in the reference cross section; the pseudo-ground-truth data acquisition unit acquires, as the pseudo-ground-truth data, information about the shape of the predetermined portion of the object in the pseudo reference cross section image; 2. The image processing device according to configuration 1, (Configuration 3) The method further includes a determination unit that determines whether the pseudo-teacher data is data in which information of the object in the pseudo reference cross section image is appropriately estimated, for each of the pseudo-teacher data, The pseudo teacher data included in the pseudo teacher data is the pseudo teacher data that has been determined by the determination unit to be information on the object that has been appropriately estimated. 3. The image processing device according to configuration 1 or 2. (Configuration 4) the information about the object is the positions of contour points of the object; The determination unit determines whether the pseudo ground truth data is data that appropriately estimates information about the object in the pseudo reference cross section image, based on pixel values ​​at positions of contour points of the object indicated by the pseudo ground truth data. 4. The image processing device according to configuration 3. (Configuration 5) the information about the object is the positions of contour points of the object; The determination unit uses a classifier trained to estimate likelihoods of contours of the object using positions of contour points of the object as input, and determines that the pseudo ground truth data is data in which information of the object in the pseudo reference cross section image has been appropriately estimated when the positions of contour points of the object indicated by the pseudo ground truth data are equal to or greater than a predetermined likelihood. 4. The image processing device according to configuration 3. (Configuration 6) a display unit that displays information about the object indicated by the pseudo-ground truth data; an input unit that receives an input from a user regarding information about the object displayed on the display unit; and The determination unit determines whether the pseudo-ground truth data is data obtained by appropriately estimating information about the object in the pseudo reference cross section image, based on the input from the user received by the input unit. 6. The image processing device according to any one of configurations 3 to 5. (Configuration 7) 7. The image processing device according to any one of configurations 1 to 6, wherein the pseudo image acquisition unit acquires, as the pseudo reference cross section image, a cross section image that intersects with an imaging plane of the three-dimensional imaging probe in the reference cross section image. (Configuration 8) 7. The image processing device according to any one of configurations 1 to 6, wherein the pseudo image acquisition unit acquires, as the pseudo reference cross section image, an image of a cross section that is different in position or orientation from the reference cross section. (Configuration 9) 7. The image processing device according to any one of configurations 1 to 6, wherein the pseudo image acquisition unit acquires, as the pseudo reference cross section image, an image of a cross section having a normal direction different from that of the reference cross section. (Configuration 10) 10. The image processing device according to any one of configurations 1 to 9, wherein the first learning unit learns the first learning model using deep learning so as to input the reference cross-sectional image and output information about the object. (Configuration 11) 11. The image processing device according to any one of configurations 1 to 10, wherein the second learning unit learns the second learning model using deep learning so as to input the reference cross-sectional image and output information about the object. (Configuration 12) The image processing device of any one of configurations 1 to 11, characterized in that the second learning unit learns the second learning model using second pseudo teacher data that is acquired by a method different from the method for acquiring the pseudo teacher data and has a smaller influence on the learning of the second learning model by the second learning unit than the pseudo teacher data. (Configuration 13) The image processing device according to any one of configurations 1 to 12, wherein the pseudo image acquisition unit acquires, as the pseudo reference cross section image, an image including a cross section image of a cross section other than a cross section obtained by rotating the orientation of the reference cross section around an axis corresponding to the irradiation direction of the ultrasound beam irradiated from the three-dimensional imaging probe. (Configuration 14) 14. The image processing device according to any one of configurations 1 to 13, further comprising a three-dimensional image acquisition unit that acquires the plurality of three-dimensional images of the object by using the three-dimensional imaging probe. (Configuration 15) the teacher data acquisition unit acquires a teacher data set consisting of a plurality of the teacher data, The first learning unit learns the first learning model using the teacher dataset to obtain the first trained model. 15. The image processing device according to any one of configurations 1 to 14. (Configuration 16) Further, a pseudo teacher data acquisition unit is provided for acquiring a pseudo teacher data set composed of a plurality of the pseudo teacher data, The second learning unit learns the second learning model using the pseudo teacher dataset to obtain the second trained model. 16. The image processing device according to any one of configurations 1 to 15. (Configuration 17) a trained model acquisition unit that uses teacher data including reference cross-sectional images acquired from a three-dimensional image including an object and ground truth data that is information about the object to train a first training model that estimates information about the object from cross-sectional images in the three-dimensional image to acquire a first trained model, acquires pseudo reference cross-sectional images that intersect with an imaging plane of the three-dimensional imaging probe in the three-dimensional image and are different from the reference cross section, acquires information about the object estimated using the pseudo reference cross-sectional images and the first trained model as pseudo ground truth data, and trains a second training model using pseudo teacher data including the pseudo reference cross-sectional images and the pseudo ground truth data to acquire a second trained model that estimates information about the object from cross-sectional images in the three-dimensional image; an input image acquisition unit that acquires a reference cross section as an input image from a three-dimensional image of the object captured using the three-dimensional imaging probe; an estimation unit that estimates information about the object using the input image and the second trained model; 1. An image processing device comprising: (Configuration 18) an image acquisition unit that acquires a reference cross-sectional image from each of a plurality of three-dimensional images including the object; a training data acquisition unit that acquires, for each of the reference cross section images, training data including the reference cross section image and correct answer data that is information about the object in the reference cross section image; a pseudo image acquisition unit that acquires, for each of the plurality of three-dimensional images, a cross section different from the reference cross section as a pseudo reference cross section image based on a relationship between an imaging plane of a three-dimensional imaging probe that images the object in the three-dimensional image and a reference cross section of the reference cross section image; a pseudo teacher data acquiring unit that acquires pseudo teacher data including the pseudo reference cross section image and the correct answer data corresponding to the reference cross section image used for acquiring the pseudo reference cross section image by the pseudo image acquiring unit; a learning unit that uses the pseudo-teacher data to learn a learning model that estimates information about the object from a cross-sectional image in the three-dimensional image; 1. An image processing device comprising: (Method 1) an image acquisition step of acquiring a reference cross-sectional image from each of a plurality of three-dimensional images including the object; a training data acquisition step of acquiring training data including the reference cross section image and correct answer data that is information on the object in the reference cross section image; a first learning step of learning a first model that estimates information about the object from a cross-sectional image in the three-dimensional image using the training data to acquire a first learned model; a pseudo image acquisition step of acquiring, for each of the plurality of three-dimensional images, a cross section different from the reference cross section as a pseudo reference cross section image based on a relationship between an imaging plane of a three-dimensional imaging probe that images the object in the three-dimensional image and a reference cross section of the reference cross section image; a pseudo-ground truth data acquisition step of acquiring pseudo-ground truth data, which is information about the object in the pseudo reference cross section image, using the first trained model; a second learning step of learning a second learning model that estimates information about the object from the cross-sectional image in the three-dimensional image using pseudo teacher data including the pseudo reference cross-sectional image and the pseudo ground truth data to acquire a second learned model; An image processing method comprising: (Method 2) A first learned model is acquired by learning a first learned model that estimates information about the object from a cross-sectional image in a three-dimensional image using training data that includes a reference cross-sectional image acquired from a three-dimensional image including the object and correct answer data that is information about the object, and a first learned model is acquired by using the first learned model to estimate information about the object from a cross-sectional image in the three-dimensional image. a trained model acquisition step of acquiring a pseudo reference cross section image that intersects with an imaging plane of the three-dimensional imaging probe in the image and is different from the reference cross section, acquiring information about the object estimated using the pseudo reference cross section image and the first trained model as pseudo ground truth data, and training a second trained model using pseudo teacher data including the pseudo reference cross section image and the pseudo ground truth data to acquire a second trained model that estimates information about the object from a cross section image in the three-dimensional image; an input image acquisition step of acquiring a reference cross section as an input image from a three-dimensional image of the object captured using the three-dimensional imaging probe; an estimation step of estimating information about the object using the input image and the second trained model; An image processing method comprising: (Method 3) an image acquisition step of acquiring a reference cross-sectional image from each of a plurality of three-dimensional images including the object; a training data acquisition step of acquiring, for each of the reference cross section images, training data including the reference cross section image and correct answer data that is information on the object in the reference cross section image; a pseudo image acquisition step of acquiring, for each of the plurality of three-dimensional images, a cross section different from the reference cross section as a pseudo reference cross section image based on a relationship between an imaging plane of a three-dimensional imaging probe that images the object in the three-dimensional image and a reference cross section of the reference cross section image; a pseudo teacher data acquiring step of acquiring pseudo teacher data including the pseudo reference cross section image and the correct answer data corresponding to the reference cross section image used to acquire the pseudo reference cross section image in the pseudo image acquiring step; a learning step of learning a learning model that estimates information about the object from a cross-sectional image in the three-dimensional image using the pseudo-teacher data; An image processing method comprising: (program) A program for causing a computer to execute each step of the image processing method according to any one of Methods 1 to 3. [Explanation of symbols]

[0107] 10 Image processing device, 40 Control unit, 43 Teacher data set acquisition unit, 44 Teacher model learning unit, 45 Pseudo reference cross section image acquisition unit, 46 Pseudo correct answer data acquisition unit, 47 Pseudo teacher data set acquisition unit, 48 Student model learning unit

Claims

1. an image acquisition unit that acquires a reference cross section image from each of a plurality of three-dimensional images including the object; a training data acquisition unit that acquires training data including the reference cross-sectional image and correct answer data that is information about the object in the reference cross-sectional image; a first learning unit that uses the teacher data to learn a first learning model that estimates information about the object from a cross-sectional image in the three-dimensional image, and acquires a first learned model; a pseudo image acquisition unit that acquires, for each of the plurality of three-dimensional images, a cross section different from the reference cross section as a pseudo reference cross section image based on a relationship between an imaging plane of a three-dimensional imaging probe that images the object in the three-dimensional image and a reference cross section of the reference cross section image; a pseudo-ground-truth data acquisition unit that acquires pseudo-ground-truth data, which is information about the object in the pseudo reference cross section image, using the first trained model; a second learning unit that learns a second learning model that estimates information about the object from a cross-sectional image in the three-dimensional image using pseudo teacher data including the pseudo reference cross-sectional image and the pseudo ground truth data, thereby acquiring a second learned model; 1. An image processing device comprising:

2. the teacher data acquisition unit acquires, as the information about the object, information about the shape of a predetermined portion of the object in the reference cross section; the pseudo-ground-truth data acquisition unit acquires, as the pseudo-ground-truth data, information about the shape of the predetermined portion of the object in the pseudo reference cross section image; 2. The image processing device according to claim 1, wherein:

3. The method further includes a determination unit that determines whether the pseudo-teacher data is data in which information of the object in the pseudo reference cross section image is appropriately estimated, for each of the pseudo-teacher data, The pseudo teacher data included in the pseudo teacher data is the pseudo teacher data that has been determined by the determination unit to be information on the object that has been appropriately estimated.

2. The image processing device according to claim 1, wherein:

4. the information about the object is the positions of contour points of the object; The determination unit determines whether the pseudo ground truth data is data that appropriately estimates information about the object in the pseudo reference cross section image, based on pixel values ​​at positions of contour points of the object indicated by the pseudo ground truth data.

4. The image processing device according to claim 3.

5. the information about the object is the positions of contour points of the object; The determination unit uses a classifier trained to estimate likelihoods of contours of the object using positions of contour points of the object as input, and determines that the pseudo ground truth data is data in which information of the object in the pseudo reference cross section image has been appropriately estimated when the positions of contour points of the object indicated by the pseudo ground truth data are equal to or greater than a predetermined likelihood.

4. The image processing device according to claim 3.

6. a display unit that displays information about the object indicated by the pseudo-ground truth data; an input unit that receives an input from a user regarding information about the object displayed on the display unit; and The determination unit determines whether the pseudo-ground truth data is a data set that indicates that information about the object in the pseudo reference cross section image has been appropriately estimated, based on the input from the user received by the input unit. Determine whether it is data or not 4. The image processing device according to claim 3.

7. The image processing apparatus according to claim 1 , wherein the pseudo image acquisition unit acquires, as the pseudo reference cross section image, a cross section image that intersects with an imaging plane of the three-dimensional imaging probe in the reference cross section image.

8. The image processing apparatus according to claim 1 , wherein the pseudo image acquisition unit acquires, as the pseudo reference cross section image, an image of a cross section that is different in position or orientation from the reference cross section.

9. The image processing apparatus according to claim 1 , wherein the pseudo image acquisition unit acquires, as the pseudo reference cross section image, an image of a cross section having a normal direction different from that of the reference cross section.

10. The image processing device according to claim 1 , wherein the first learning unit learns the first learning model using deep learning so as to input the reference cross-sectional image and output information about the object.

11. The image processing device according to claim 1 , wherein the second learning unit learns the second learning model using deep learning so as to input the reference cross-sectional image and output information about the object.

12. 2. The image processing device according to claim 1, wherein the second learning unit learns the second learning model using second pseudo teacher data that is acquired by a method different from that of acquiring the pseudo teacher data and has a smaller influence on learning of the second learning model by the second learning unit than the pseudo teacher data.

13. The image processing device according to claim 1, characterized in that the pseudo image acquisition unit acquires, as the pseudo reference cross section image, an image including a cross section image of a cross section other than a cross section obtained by rotating the orientation of the reference cross section around an axis corresponding to the irradiation direction of the ultrasound beam emitted from the three-dimensional imaging probe.

14. 2. The image processing apparatus according to claim 1, further comprising a three-dimensional image acquisition unit for acquiring the plurality of three-dimensional images of the object by using the three-dimensional imaging probe.

15. the teacher data acquisition unit acquires a teacher data set consisting of a plurality of the teacher data, The first learning unit learns the first learning model using the teacher dataset to obtain the first trained model.

2. The image processing device according to claim 1, wherein:

16. Further, a pseudo teacher data acquisition unit is provided for acquiring a pseudo teacher data set composed of a plurality of the pseudo teacher data, The second learning unit learns the second learning model using the pseudo teacher dataset to obtain the second learned model.

2. The image processing device according to claim 1, wherein:

17. A first learned model is learned by using teacher data including a reference cross-sectional image acquired from a three-dimensional image including an object and correct answer data that is information about the object, and the first learned model is acquired by estimating information about the object from a cross-sectional image in the three-dimensional image, and a pseudo-reference cross-section that intersects with an imaging plane in the three-dimensional image by the three-dimensional imaging probe and is different from the reference cross-section. a trained model acquisition unit that acquires quasi-sectional images, acquires information about the object estimated using the pseudo reference cross-sectional images and the first trained model as pseudo-ground-truth data, trains a second trained model using pseudo teacher data including the pseudo reference cross-sectional images and the pseudo-ground-truth data, and acquires a second trained model that estimates information about the object from cross-sectional images in the three-dimensional image; an input image acquisition unit that acquires a reference cross section as an input image from a three-dimensional image of the object captured using the three-dimensional imaging probe; an estimation unit that estimates information about the object using the input image and the second trained model; 1. An image processing device comprising:

18. an image acquisition unit that acquires a reference cross section image from each of a plurality of three-dimensional images including the object; a training data acquisition unit that acquires, for each of the reference cross section images, training data including the reference cross section image and correct answer data that is information about the object in the reference cross section image; a pseudo image acquisition unit that acquires, for each of the plurality of three-dimensional images, a cross section different from the reference cross section as a pseudo reference cross section image based on a relationship between an imaging plane of a three-dimensional imaging probe that images the object in the three-dimensional image and a reference cross section of the reference cross section image; a pseudo teacher data acquiring unit that acquires pseudo teacher data including the pseudo reference cross section image and the correct answer data corresponding to the reference cross section image used for acquiring the pseudo reference cross section image by the pseudo image acquiring unit; a learning unit that uses the pseudo-teacher data to learn a learning model that estimates information about the object from a cross-sectional image in the three-dimensional image; 1. An image processing device comprising:

19. an image acquiring step of acquiring a reference cross section image from each of a plurality of three-dimensional images including the object; a training data acquisition step of acquiring training data including the reference cross section image and correct answer data that is information on the object in the reference cross section image; a first learning step of learning a first model that estimates information about the object from a cross-sectional image in the three-dimensional image using the training data to acquire a first learned model; a pseudo image acquisition step of acquiring, for each of the plurality of three-dimensional images, a cross section different from the reference cross section as a pseudo reference cross section image based on a relationship between an imaging plane of a three-dimensional imaging probe that images the object in the three-dimensional image and a reference cross section of the reference cross section image; a pseudo-ground-truth data acquisition step of acquiring pseudo-ground-truth data, which is information about the object in the pseudo reference cross section image, using the first trained model; a second learning step of learning a second learning model that estimates information about the object from the cross-sectional image in the three-dimensional image using pseudo teacher data including the pseudo reference cross-sectional image and the pseudo ground truth data, to acquire a second learned model; An image processing method comprising:

20. a trained model acquisition step of using teacher data including a reference cross section image acquired from a three-dimensional image including an object and ground truth data that is information about the object to train a first learning model that estimates information about the object from a cross-sectional image in the three-dimensional image to acquire a first trained model, acquiring pseudo reference cross-sectional images that intersect with an imaging plane of the three-dimensional imaging probe in the three-dimensional image and differ from the reference cross section, acquiring information about the object estimated using the pseudo reference cross-sectional image and the first trained model as pseudo ground truth data, and training a second learning model using pseudo teacher data including the pseudo reference cross-sectional image and the pseudo ground truth data to acquire a second trained model that estimates information about the object from a cross-sectional image in the three-dimensional image; an input image acquisition step of acquiring a reference cross section as an input image from a three-dimensional image of the object captured using the three-dimensional imaging probe; an estimation step of estimating information about the object using the input image and the second trained model; An image processing method comprising:

21. an image acquiring step of acquiring a reference cross section image from each of a plurality of three-dimensional images including the object; a training data acquisition step of acquiring, for each of the reference cross section images, training data including the reference cross section image and correct answer data that is information on the object in the reference cross section image; a pseudo image acquisition step of acquiring, for each of the plurality of three-dimensional images, a cross section different from the reference cross section as a pseudo reference cross section image based on a relationship between an imaging plane of a three-dimensional imaging probe that images the object in the three-dimensional image and a reference cross section of the reference cross section image; a pseudo teacher data acquiring step of acquiring pseudo teacher data including the pseudo reference cross section image and the correct answer data corresponding to the reference cross section image used to acquire the pseudo reference cross section image in the pseudo image acquiring step; a learning step of learning a learning model that estimates information about the object from a cross-sectional image in the three-dimensional image using the pseudo-teacher data; An image processing method comprising:

22. A program for causing a computer to execute each step of the image processing method according to any one of claims 19 to 21.

Images