Learning device, image processing device, medical image capturing device, learning method and program

Abstract

To provide a technique that can perform contour estimation with high accuracy.SOLUTION: A learning device acquires a learned model in which a contour of an object in a picked-up image of the object is learned on the basis of positions of predetermined feature points, and comprises: a learning data acquisition unit that acquires the position of the contour of the object in a plurality of images of a case and the positions of the feature points of the object in the images as learning data; a normalization unit that creates a normalized image and a normalized contour through normalization processing of converting the coordinates of the image and the contour into a normalization space so as to bring the feature points in the plurality of images of a case to the same positions; a deformation unit that, in a state in which the positions of the feature points in the normalization space are fixed to the same positions, creates a deformed image and a deformed contour obtained by applying deformation to the normalized image and the normalized contour; and a model acquisition unit that acquires the learned model on the basis of, the deformed image and the deformed contour.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a learning device, an image processing device, a medical image capturing device, a learning method, and a program. [Background technology]

[0002] In the medical field, diagnoses are made using images acquired by various imaging diagnostic devices (modalities) such as ultrasound imaging diagnostic devices. In these diagnoses, information such as the area and volume of a region of interest shown in an image is used for diagnosis. In order to calculate the area of ​​a region, it is necessary to estimate contour information, which is information that represents the contour shape of the region, from the image. When this contour estimation work is performed manually, it imposes a burden on operators such as doctors, and therefore various techniques for performing contour estimation have been proposed to reduce the operator's labor.

[0003] For example, Patent Document 1 discloses a technique for estimating contours using points (input points) manually input by an operator, in which contour estimation is performed based on an initial contour connecting the input points, using a two-dimensional ultrasound image of the cardiac region as the target. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-73798 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technique described in Patent Document 1, the accuracy of contour estimation depends on the accuracy of input points input by the operator, so that there may be cases in which an appropriate shape cannot be estimated.

[0006] In order to solve the above-mentioned problems, the present invention provides a technique capable of performing highly accurate contour estimation. [Means for solving the problem]

[0007] A learning device according to one aspect of the present invention has the following configuration: but A learning device that acquires a trained model trained based on the positions of predetermined feature points, a learning data acquisition unit that acquires, as learning data, positions of contours of objects in images of a plurality of cases and positions of feature points of the objects in the images; a normalization unit that generates normalized images and normalized contours by a normalization process that converts the images and the contours into a normalized space so that the feature points in the images of the plurality of cases are at the same positions; a deformation unit that generates a deformed image and a deformed contour by applying deformation to the normalized image and the normalized contour while fixing the positions of the feature points at the same positions in the normalized space; a model acquisition unit that acquires the trained model based on the deformed image and the deformed contour. 、 The deformation unit generates the deformed image and the deformed contour by adding a deviation value that translates a position in the normalized space, and limits the addition of the deviation value when the deviation value exceeds a predetermined allowable range. do. A learning device according to another aspect of the present invention has the following configuration: A learning device for acquiring a trained model in which a contour of an object in an image of the object is trained based on positions of predetermined feature points, a learning data acquisition unit that acquires, as learning data, positions of contours of objects in images of a plurality of cases and positions of feature points of the objects in the images; a normalization unit that generates a normalized image and a normalized contour by a normalization process that converts the image and the contour into a normalized space in coordinates so that new feature points to which a variation value has been added at a position of the feature point are positioned in the same position in the images of the plurality of cases; a model acquisition unit that acquires the trained model based on the normalized image and the normalized contour; Equipped with The normalization unit classifies the feature points in the normalized space into fixed feature points whose positions are fixed and variation-permissible feature points whose positions are permitted to vary by adding the variation value. A learning method according to another aspect of the present invention has the following configuration: A learning method for a learning device that acquires a trained model in which a contour of a predetermined object in an image captured of the object is trained based on positions of predetermined feature points, comprising: a learning data acquiring step of acquiring, as learning data, positions of contours of objects in images of a plurality of cases and positions of feature points of the objects in the images; a normalization step of generating normalized images and normalized contours by a normalization process of converting the images and the contour positions into a normalized space so that the feature points in the images of the plurality of cases are at the same positions; a deformation step of generating a deformed image and a deformed contour by applying deformation to the normalized image and the normalized contour while fixing the positions of the feature points in the normalized space; A model acquisition step of acquiring the trained model based on the deformed image and the deformed contour; having The deformation step generates the deformed image and the deformed contour by adding a deviation value that translates a position in the normalized space, and limits the addition of the deviation value when the deviation value exceeds a predetermined allowable range. A learning method according to another aspect of the present invention includes the following steps: but A learning method for acquiring a trained model trained based on positions of predetermined feature points, comprising: a learning data acquiring step of acquiring, as learning data, positions of contours of objects in images of a plurality of cases and positions of feature points of the objects in the images; a normalization step of generating a normalized image and a normalized contour by a normalization process of converting the image and the contour into a normalized space in coordinates so that the new feature points to which the variation values ​​have been added are positioned in the same positions as the new feature points to which the variation values ​​have been added in the images of the plurality of cases; A model acquisition step of acquiring the trained model based on the normalized image and the normalized contour, The normalization step classifies the feature points in the normalized space into fixed feature points whose positions are fixed and variation-tolerant feature points whose positions are allowed to vary by adding the variation value. Effect of the Invention

[0008] According to the present invention, it is possible to perform contour estimation with high accuracy. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing the functional arrangement of an image processing apparatus according to first to fourth embodiments. [Diagram 2] 4 is a flowchart showing an example of a processing procedure of the image processing apparatus according to the first to third embodiments. [Diagram 3] 1 is a flowchart showing an example of a procedure for acquiring a learning model according to the first to third embodiments. [Figure 4] 1A is a diagram for explaining the process of transforming coordinates into a normalized space (normalization process); FIG. 1B is a diagram showing the process of augmenting learning data in learning; [Diagram 5] FIG. 4 is a diagram showing the flow of a contour estimation process. [Figure 6] 13 is a flowchart showing an example of a processing procedure of an image processing apparatus according to a fourth embodiment. [Figure 7] 13 is a flowchart showing an example of a procedure for acquiring a learning model according to the fourth embodiment. [Figure 8] 13 is a flowchart showing a modified example of the learning model acquisition process according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] A medical imaging apparatus according to an embodiment of the present invention has an image processing device for processing an image, and the image processing device provides a function of estimating (extracting) a contour of a subject from an input image and feature points (hereinafter, input points) input by a user on the input image. The input image to be processed is a medical image, that is, an image of a subject (such as a human body) captured or generated for the purpose of medical diagnosis, examination, research, etc., and is typically an image acquired by an imaging system called a modality.

[0012] For example, ultrasound images obtained by an ultrasound diagnostic device, X-ray CT images obtained by an X-ray CT device, MRI images obtained by an MRI device, etc. can be processed. The input image may be a two-dimensional image or a three-dimensional image, and may be an image of one time phase or an image of multiple time phases. The region of interest (object) is a part of the image, such as an anatomical structure (organs, blood vessels, bones, etc.) or a lesion. What is selected as the region of interest (object) can be set arbitrarily.

[0013] In the following description, a specific example of an image processing device according to an embodiment of the present invention will be described in detail, taking as an example a case in which a contour of the right ventricle of the heart is estimated as an object from a two-dimensional ultrasound image.

[0014] <First embodiment> In the first embodiment, contour estimation using input points by a user for contour estimation is referred to as semi-automatic contour estimation in this embodiment. A configuration is described in which contour estimation of an object is performed by semi-automatic estimation in a normalized space that has been coordinate-transformed using the coordinate values ​​of specific feature points (input points) input on an input image. In this embodiment, contour estimation of a region of interest performed by an image processing device (or system) based on feature points manually input by a user is referred to as semi-automatic estimation. The image processing device described in this embodiment performs contour estimation of an object by semi-automatic estimation using feature points (input points) manually input by a user.

[0015] The image processing device according to the first embodiment has a function of automatically estimating (extracting) the contour (contour information) of the right ventricle, which is an object, from an input image and the coordinates of predetermined feature points (input points) input by a user on the input image. Here, the contour (contour information) of the object is represented by the coordinates of a predetermined number (hereinafter, Np) of points (contour point group) arranged on the contour.

[0016] The image processing device of this embodiment estimates the contour (contour information) of an object based on a trained model (statistical model) constructed from training data by statistical processing. Here, the training data is composed of images (samples) of a plurality of cases in which the correct answer for the contour information of the object is known. In addition, the coordinates of predetermined feature points are given to each image of the training data. The trained model is constructed in a normalized space in which each image of the training data is coordinate-converted so that the feature points (for example, for each type of feature point indicating the apex position of the cardiac cavity) are located at the same predetermined coordinates. Then, the contour estimation of the input image is performed after performing a similar normalization process using the input points.

[0017] In this way, the image processing device of this embodiment can estimate the contour of the right ventricle based on the statistical tendency of the trained model in a state where the coordinate values ​​of the input points are fixed in the normalized space (a state where the coordinate values ​​are transformed to the same predetermined coordinates). Note that, in this embodiment, a configuration for estimating contour information of the right ventricle will be described as an example of an object, but the object is not limited to the right ventricle, and can also be applied to the case of estimating the contour of other parts including the left ventricle, left atrium, and right atrium, or a region that combines a plurality of these regions.

[0018] The functional configuration of the image processing device 10 of this embodiment and the processing executed by the image processing device 10 will be described below with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of an image processing system 100 (also called a medical image processing system) including the image processing device 10 of this embodiment. The image processing system 100 includes the image processing device 10 and a database 22. The image processing device 10 is connected to the database 22 via a network 21 in a communicable state. The network 21 includes, for example, a LAN (Local Area Network) or a WAN (Wide Area Network).

[0019] The database 22 holds and manages a plurality of images and information. The information managed by the database 22 includes an input image provided for contour estimation processing by the image processing device 10 and learning data for generating a learned model. Here, the information managed by the database 22 may include information on a learned model generated from the learning data instead of the learning data. The information on the learned model may be stored in an internal memory (ROM 32 or storage unit 34) of the image processing device 10 instead of the database 22. The image processing device 10 is capable of acquiring the data held in the database 22 via the network 21.

[0020] The image processing device 10 includes a communication IF (Interface) 31 (communication unit), a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33, a storage unit 34, an operation unit 35, a display unit 36, and a control unit 37.

[0021] The communication IF 31 (communication unit) is configured with a LAN card or the like, and realizes communication between an external device (e.g., the database 22, etc.) and the image processing device 10. The ROM 32 is configured with a non-volatile memory or the like, and stores various programs and various data. The RAM 33 is configured with a volatile memory or the like, and is used as a work memory for temporarily storing programs and data being executed. The storage unit 34 is configured with a HDD (Hard Disk Drive) or the like, and stores various programs and various data.

[0022] The operation unit 35 is composed of a keyboard, a mouse, a touch panel, etc., and inputs instructions from a user (for example, a doctor or a medical technician) to various devices.

[0023] The display unit 36 ​​is configured with a display and the like, and displays various information to the user.

[0024] The control unit 37 is configured with a CPU (Central Processing Unit) and the like, and controls the processing in the image processing device 10. The control unit 37 has, as its functional configuration, an image acquisition unit 41, a learning model acquisition unit 43, a contour estimation unit 44, and a display processing unit 45. The control unit 37 may also have a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), or the like.

[0025] (Image acquisition unit 41) The image acquisition unit 41 acquires an input image (an image with unknown contour information) to be processed from the database 22. The input image is an image of a subject acquired by various modalities, and in this embodiment, it is a two-dimensional ultrasound image of the heart. The image acquisition unit 41 may acquire the input image directly from the modality. In this case, the image processing device 10 may be implemented in the console of the modality (imaging system). In this embodiment, an example in which the input image is a two-dimensional ultrasound image will be described, but other types of images may also be used. The method of this embodiment is applicable to images that are two or more dimensional (such as multiple two-dimensional images, two-dimensional moving images, three-dimensional still images, multiple three-dimensional images, or three-dimensional moving images). In addition, it is applicable regardless of the type of modality.

[0026] (Position acquisition unit 42) The position acquisition unit 42 acquires, as input points, feature points (positions of feature points) input by the user for the input image. The position acquisition unit 42 acquires coordinate information (position information) of a predetermined feature point (input point) that is manually input by the user via the operation unit 35 on the input image. In this embodiment, two points are used as the predetermined feature points: the apex position of the right ventricle and the midpoint between the left and right annulus positions of the tricuspid valve. The function of the position acquisition unit 42 will be described later in the explanation of step S202.

[0027] (Learning model acquisition unit 43) The learning model acquisition unit 43 has a function of learning the contour of an object in an image of the object and acquiring a learned model, as well as a function of acquiring (selecting) a desired learned model from learned models preregistered in a storage unit of the image processing device or multiple learned models stored externally. The learning model acquisition unit 43 acquires a learned model in which the contour of an object in an image of the object is learned based on the positions of predetermined feature points. Specifically, the learning model acquisition unit 43 acquires a learned model that has learned the relationship between an image of a predetermined object and the contour of the object in the image, and that is generated in a normalized space in which the positions of predetermined feature points of the object are aligned.

[0028] The learning model acquisition unit 43 acquires learning data from the database 22 and performs statistical analysis on the learning data. Each of the samples constituting the learning data includes data including pixel value information of images of multiple learning cases, contour information of objects in the learning images, and coordinate information of predetermined feature points. Specifically, the learning model acquisition unit 43 functions as a learning data acquisition unit and acquires images of multiple cases, the positions of the contours of objects in the images of multiple cases (positions of contour lines), and the positions of feature points of objects in the images of multiple cases as learning data. Here, the feature points in the learning data represent points that are anatomically identical to the feature points acquired by the position acquisition unit 42. That is, when the position acquisition unit 42 acquires two points, the apex position of the right ventricle and the midpoints of the left and right annular positions of the tricuspid valve as feature points, the feature points in the samples constituting the learning data are also two points, the apex position of the right ventricle and the midpoints of the left and right annular positions of the tricuspid valve.

[0029] Then, the learning model acquisition unit 43 acquires a learned model (for example, predetermined information constituting a subspace representing the statistical distribution of a sample group) from the result of the statistical analysis. At this time, the learned model is defined in a normalized space that has been coordinate-transformed so that the coordinates of the feature points in each sample are the same (for each type of feature point). The learning model acquisition unit 43 generates a normalized image and a normalized contour by a normalization process that transforms the image and contour of the learning data into the normalized space in coordinates so that the feature points in the images of multiple cases are at the same position. The learning model acquisition unit 43 functions as a normalization unit and acquires a normalized image obtained by transforming the input image into a normalized space so that the position of the input point is aligned with the position of the feature point of the learned model. Specifically, the learning model acquisition unit 43 acquires a normalized image obtained by transforming the input image into a normalized space in coordinates so that the position of the input point is aligned with the position of the feature point of the learned model.

[0030] By doing this, when contour estimation is performed using this trained model, contour information can be estimated based on the statistical distribution of the sample group while the coordinates of the feature points (input points) acquired by the position acquisition unit 42 do not change (i.e., are fixed).

[0031] (Contour Estimation Unit 44) The contour estimation unit 44 estimates the contour (contour information) of an object in an input image using the normalized image and the trained model. Here, the contour (contour information) of an object includes the contour line and position information of the contour points constituting the contour line. The contour estimation unit 44 estimates contour information of an object (e.g., the right ventricle) in an input image from the input image acquired by the image acquisition unit 41 and the coordinates of feature points (input points) set on the input image acquired by the position acquisition unit 42 using the trained model acquired by the trained model acquisition unit 43.

[0032] The contour estimation unit 44 acquires a normalized image obtained by performing coordinate conversion on the input image so as to match the position of the input point with the position of the feature point of the learned model. Then, the contour estimation unit 44 estimates the contour of the object in the input image using the normalized image and the learned model. As a specific process, the contour estimation unit 44 performs coordinate conversion on the input image into a normalized space that is the same as the normalized space in which the learned model was generated, using information on the coordinate values ​​of the feature point (input point). Then, after estimating the contour information of the object (right ventricle) in the normalized space, the contour information in the original space is calculated by applying an inverse conversion of the coordinate conversion. Note that in this embodiment, in the normalized space, the pixel value information of the input image is set as known information, and the contour information of the object in the input image is set as unknown information, and the contour information is estimated from the pixel value information of the input image and the learned model by a matrix operation using the Back projection for lost pixels (BPLP) method. Details of BPLP will be described later.

[0033] (Display processing unit 45) Based on the result calculated by the contour estimation unit 44, the display processing unit 45 displays the input image and the estimated contour information of the object in the image display area of ​​the display unit 36 ​​in a display form that allows easy visual recognition.

[0034] Each of the components of the image processing device 10 functions according to a computer program. For example, the control unit 37 (CPU) reads and executes a computer program stored in the ROM 32 or the storage unit 34 using the RAM 33 as a work area, thereby realizing the function of each component. Note that some or all of the functions of the components of the image processing device 10 may be realized by using a dedicated circuit. Also, some of the functions of the components of the control unit 37 may be realized by using a cloud computer. For example, a calculation device located at a location different from the image processing device 10 may be communicably connected to the image processing device 10 via the network 21, and the image processing device 10 and the calculation device may transmit and receive data, thereby realizing the function of the components of the image processing device 10 or the control unit 37.

[0035] Next, an example of a processing procedure of the image processing device 10 in FIG. 1 will be described with reference to the flowchart in FIG.

[0036] (Step S201: Acquire and display image) In step S201, when the user issues an instruction to acquire an image via the operation unit 35, the image acquisition unit 41 acquires an input image designated by the user from the database 22 and stores it in the RAM 33. At this time, the display processing unit 45 displays the input image in the image display area of ​​the display unit 36. An example of the input image is shown in FIG. 5(a).

[0037] The input image may be specified by any known method. For example, the user may directly specify an image on the database 22 via the operation unit 35, or the image acquisition unit 41 may automatically select an image that satisfies a predetermined condition as an input image from a group of images specified by the user. In addition to obtaining the input image from the database 22, the image acquisition unit 41 may obtain the input image from images obtained by various image diagnostic devices (modalities), for example, from ultrasound images captured by an ultrasound diagnostic imaging device from time to time. For example, the latest image captured at the time when the user instructs image acquisition can be used as the input image. In addition, based on information such as the cardiac potential associated with the image, an image of a predetermined time phase (for example, end systole or end diastole) captured after the time when the user instructs acquisition may be automatically determined, and the image may be used as the input image. (Step S202: Acquire feature point coordinates) In step S202, when the user inputs coordinate information of a specific feature point on the input image displayed in the display area of ​​the display unit 36 ​​via the operation unit 35, the position acquisition unit 42 acquires the coordinate information (position information) of the feature point (input point) manually input by the user on the input image, and stores it in the RAM 33.

[0038] 5(a) shows an example of coordinate information of feature points (input points). The feature points acquired in this embodiment are two points, the apex position 501 and the midpoint position 503 of the left and right valve annulus. Note that the apex position 501 and the midpoint position 503 of the left and right valve annulus are merely exemplary, and the two feature points for which coordinate information is acquired may be other than the combination of the apex and the midpoint of the left and right valve annulus. For example, the two points, the position of the right annulus of the tricuspid valve and the position of the left annulus, may be used as feature points.

[0039] Furthermore, the number of feature points is not limited to two, and may be one point at the apex position, or three points at the positions of the apex position and the left and right valve annulus.Four or more points including positions on the contour line between the apex position and the left and right valve annulus may be used.Based on an input from a user, the position acquisition unit 42 can acquire coordinate information of feature points required for normalization processing of the input image (step S204) described later.

[0040] Note that the coordinate information of the feature points can be acquired by various methods, and any known method may be used. For example, the user may specify the coordinate information by clicking the position of the feature point on the input image using a mouse as the operation unit 35, or may specify the coordinate information using a trackball. At this time, the user may separately specify information (label) for identifying which feature point the specified coordinate information corresponds to via a UI (user interface) (not shown).

[0041] Alternatively, the user may specify the coordinate positions of the feature points themselves via the operation unit 35, and the position acquisition unit 42 may automatically identify whether a plurality of feature points are the apex or the left and right annular midpoints based on the coordinate positions of the specified feature points. For example, the position acquisition unit 42 may acquire coordinate information regarding a feature point close to the position of the ultrasound probe (i.e., a feature point whose y coordinate value is close to 0) as the apex, and acquire coordinate information regarding a feature point far from the position of the ultrasound probe as the left and right annular midpoints. Alternatively, the order in which the user inputs the coordinate information may be fixed in advance, and the position acquisition unit 42 may acquire coordinate information regarding the first point inputted first as the apex position, and acquire coordinate information regarding the second point inputted next as the left and right annular midpoint position.

[0042] Furthermore, instead of having the user directly specify the left and right annulus midpoint positions via the operation unit 35, it is also possible to have the user specify a position that is easy to visually confirm, and the position acquisition unit 42 acquires coordinate information of the left and right annulus midpoints based on the specified position. For example, the user may specify the positions of the left and right annulus, which are easy to visually confirm, via the operation unit 35, and the position acquisition unit 42 may calculate the midpoint between the positions of the left and right annulus specified by the user, and acquire the position of the calculated midpoint as the left and right annulus midpoint positions.

[0043] The position acquisition unit 42 can also automatically detect the coordinate information of these feature points by using template matching or machine learning. The position acquisition unit 42 can also correct the automatically detected coordinate information of the feature points by user input via the operation unit 35.

[0044] Furthermore, the position acquisition unit 42 and the display processing unit 45 superimpose a marker indicating each feature point on the input image at the average position where each feature point is likely to exist, and the position acquisition unit 42 can also correct the coordinate information of each feature point at the average position based on the position of the marker dragged by the user.

[0045] (Step S203: Obtaining a trained model) In step S203, the learning model acquisition unit 43 acquires learning data from the database 22. Then, the learning model acquisition unit 43 constructs a learned model using the learning data acquired from the database 22. Note that the learning data is preferably composed of image data of a plurality of different patients in order to enhance the robustness of the learned model. However, the image data of a plurality of different patients may include images of the same patient captured at different times.

[0046] FIG. 3 is a flowchart showing an example of the procedure of a learned model acquisition process according to this embodiment. Below, the details of the process performed by the learned model acquisition unit 43 will be explained using the flowchart of FIG.

[0047] (Step S301: Obtaining learning data images and correct contours) In step S301, the learning model acquisition unit 43 acquires images of a plurality of cases (samples) as learning data and correct answers (correct contours) of contour information of an object in each sample image from the database 22. The contour information of an object includes the position (coordinates) of the contour line of the object in each image.

[0048] (Step S302: Acquire feature point coordinates of learning data) In step S302, the learning model acquisition unit 43 acquires coordinate information (position information) of a predetermined feature point set for each sample image of the learning data acquired in step S301 from the database 22. That is, the learning model acquisition unit 43 acquires the position (coordinate information (position information)) of the feature point of the object in each image from the database 22. As described above, in this embodiment, the feature points are two points, the apex of the heart and the midpoint of the left and right valve annulus.

[0049] In the above description, the feature points set for each sample of the learning data are anatomically identical to the feature points (input points) acquired in step S202. However, the feature points set for each sample of the learning data do not necessarily have to be anatomically identical to the feature points (input points).

[0050] When the feature points set for each sample of the learning data are not identical to the anatomical feature points (input points), the learning model acquisition unit 43 may calculate the coordinates of the feature points identical to the feature points (input points) by a predetermined calculation. For example, when the feature points set for the learning data are three points each of the apex, left annulus, and right annulus, the learning model acquisition unit 43 may calculate the midpoint of the left annulus position and the right annulus position, and set the calculated position as the left and right annulus midpoint position, thereby generating coordinate information of the two feature points, the apex and the left and right annulus midpoints.

[0051] Conversely, if the feature points (input points) acquired in step S202 are a total of three points, namely the apex, left annulus, and right annulus, and the feature points set for the learning data are two points, namely the apex and the midpoint of the left and right annulus, then, for example, the learning model acquisition unit 43 may calculate the respective positions of the left and right annulus by template matching starting from the midpoint of the left and right annulus.

[0052] Furthermore, when the contour information of the object in the training data includes coordinates of these feature points, the learning model acquisition unit 43 may acquire coordinate values ​​of the predetermined feature points from the contour information. Furthermore, when the contour information of the object in the training data includes points whose relative positional relationships with these feature points are known, the learning model acquisition unit 43 may calculate coordinate values ​​of the predetermined feature points from the contour information. Alternatively, the acquisition of the feature point coordinates for each sample of the training data may be performed using any known method.

[0053] (Step S303: Transformation of coordinates into normalized space) In step S303, the learning model acquisition unit 43 normalizes the pixel value information of the image of each sample of the learning data and the contour information of the object by performing coordinate transformation using the feature point coordinates acquired for each sample of the learning data in step S302. The learning model acquisition unit 43 performs coordinate transformation of the image and the contour line in the image into a normalized space based on the positions (coordinates) of the feature points in the image of the learning case, to generate a normalized image and a normalized contour. At this time, the learning model acquisition unit 43 performs coordinate transformation such that the coordinates of the feature points after the coordinate transformation are the same in all samples of the learning data (for each type of feature point). That is, all samples of the learning data are coordinate transformed into a normalized space normalized by the coordinate values ​​of the feature point coordinates.

[0054] The coordinate transformation process (normalization process) to the normalized space will be specifically described with reference to Fig. 4(a). When coordinate transformation is not performed, the positions, sizes, and inclinations of objects in each sample image of the learning data are various, but by coordinate transformation, the contour information of the object (e.g., the right ventricle) in each sample image of the learning data is transformed as shown in Fig. 4(a).

[0055] FIG. 4(a) shows an example of the result of coordinate transformation (normalization) of contour information of the right ventricle in the learning data of about 500 cases so that two points, the apex 401 and the midpoints 402 of the left and right valve annuli, coincide with each other. In FIG. 4(a), the contour information is expressed as a set of coordinate values ​​of a group of 17 contour points (i.e., Np=17), and the 17 points are displayed in different colors so that they can be identified. That is, the number of points of a specific color (e.g., black) coincides with the number of samples of the learning data (about 500). In this example, one of the contour points representing the contour information represents the apex 401, which is a feature point. In FIG. 4(a), the "x" mark indicates the midpoints 402 of the left and right valve annuli. At this time, the apex 401 is coordinate transformed into a normalized space so that it has the same coordinates in all samples, so it is displayed as one point in FIG. 4(a), but in reality, points for the number of samples overlap.

[0056] In Fig. 4(a), the horizontal axis represents the x direction, and the vertical axis represents the y direction. In other words, it can be seen that the apex 401 has coordinate values ​​(300, 100), the left and right annular midpoint 402 has coordinate values ​​(300, 400), and the coordinate transformation is performed so that the inclination of the axis (long axis) connecting the apex 401 and the left and right annular midpoint 402 coincides with the y axis. The process of transforming the coordinates using two feature points in this way is performed by the learning model acquisition unit 43 calculating a coordinate transformation matrix for each sample that aligns the position, angle, and scale, and applying the coordinate transformation matrix to each coordinate point of the pixel value information and contour information of the sample.

[0057] In the following description, each image in the learning data after the coordinate transformation has Nx pixels in the x direction and Ny pixels in the y direction. That is, the total number of pixels constituting each image in the learning data after the coordinate transformation is Nx×Ny. At this time, the number of pixels Nx in the x direction and the number of pixels Ny in the y direction are required to be large enough to contain the contour information after the coordinate transformation. In the example shown in FIG. 4(a), it can be seen that the contour information after the coordinate transformation is distributed in a range containing 450 pixels in the x direction and 550 pixels in the y direction. In this case, as the contour information after the coordinate transformation, the number of pixels Nx in the x direction is required to contain at least 450 pixels, and the number of pixels Ny in the y direction is required to contain at least 550 pixels.

[0058] In this embodiment, the number of pixels Nx in the x direction and the number of pixels Ny in the y direction are both 600 pixels, but the number of pixels Nx in the x direction and the number of pixels Ny in the y direction do not necessarily have to be the same value.

[0059] In the present embodiment, the case where the learning data is coordinate-transformed (normalized) using two feature points has been described as an example, but the learning model acquisition unit 43 can also perform normalization using feature points other than two. For example, when only one point (e.g., the apex) is used as the feature point, the learning model acquisition unit 43 performs normalization by coordinate transformation that aligns the feature point positions by translation. When three points (e.g., the positions of the apex and the left and right valve annuli) are used as the feature points, the learning model acquisition unit 43 performs normalization by affine transformation that aligns the positions of these three points. When four or more points are used as the feature points, the learning model acquisition unit 43 performs normalization by non-rigid transformation that matches the four points. As the non-rigid transformation in this case, the learning model acquisition unit 43 can use a known method such as spline transformation or FFD (Free form deformation).

[0060] When performing coordinate transformation (normalization) of learning data using two feature points, the learning model acquisition unit 43 may use the position of the main feature point (e.g., the apex of the heart) to align the positions, and may use another feature point (e.g., the midpoints of the left and right valve annuli) only to align the axial directions in the normalized space. In this case, the learning model acquisition unit 43 does not align the scale of the other feature point (the midpoints of the left and right valve annuli), but aligns the inclination of the axis connecting the main feature point (the apex of the heart) and the other feature point (the midpoints of the left and right valve annuli) to the axial direction (e.g., the y-axis direction) in the normalized space.

[0061] Similarly, when three points are used, the learning model acquisition unit 43 can apply a normalization method that aligns the positions of the main feature points (e.g., the apex of the heart) and matches the other feature points as much as possible within the range of rotation and equal-magnification transformation. In addition, the learning model acquisition unit 43 can use any known normalization method as long as it is a method that performs geometric normalization using the coordinates of a common feature point.

[0062] (Step S304: Inflation of learning data) In step S304, the learning model acquisition unit 43 performs an augmentation process (data augmentation) of the learning data in the normalized space obtained by the coordinate transformation in step S303. The learning model acquisition unit 43 functions as a transformation unit, and generates a deformed image and a deformed contour by adding a transformation (variation value) to the normalized image and the normalized contour while fixing the positions of the feature points in the normalized space at the same positions. Then, the learning model acquisition unit 43 acquires a trained model based on the deformed image and the deformed contour. The learning model acquisition unit 43 acquires a trained model based on the training data (augmented training data) in which the deformed image and the deformed contour are added to the training data.

[0063] The learning model acquisition unit 43, which functions as a deformation unit, generates a deformed image and a deformed contour by adding a variation value that translates the position in the normalized space while fixing the position of the feature point at the same position. The learning model acquisition unit 43 also generates a deformed image and a deformed contour by adding a variation value that rotates in the normalized space. The learning model acquisition unit 43 also generates a deformed image and a deformed contour by adding a variation value of a position that expands or reduces the relative distance to the position of the fixed feature point.

[0064] The processing of this step will be specifically described with reference to FIG. 4(b). FIG. 4(b) shows an average contour shape obtained by averaging the contour information of all learning data in the normalized space. In FIG. 4(b), the inner line is the contour line. The outer line is a line drawn with a predetermined width relative to the contour line, and this represents the heart wall. When the learning model acquisition unit 43 performs the padding process, if the entire contour is translated or rotated, the positions of the apex 401 and the left and right annular midpoints 402 will change. On the other hand, as shown by the arrow 403 in FIG. 4(b), if the deformation is such that the average contour shape is enlarged or reduced (scaling) in the x-axis direction around the axis connecting the apex 401 and the left and right annular midpoints 402, the positions of the apex 401 and the left and right annular midpoints 402 will not change. In this way, the learning model acquisition unit 43 performs the padding process of the learning data by deforming each sample of the learning data so that the coordinate values ​​of each feature point used in the coordinate transformation do not change.

[0065] Specifically, for each sample (pair of image and contour information) of the normalized learning data, the learning model acquisition unit 43 performs scaling processing in the x-axis direction at multiple magnifications while fixing the positions of the two feature points used for normalization (in the example shown in FIG. 4(b), the positions of the apex 401 and the midpoints 402 of the left and right annulus). For example, the learning model acquisition unit 43 performs scaling processing on each sample at multiple magnifications (for example, 0.8, 0.9, 1.1, 1.2, etc.) to generate N pieces of inflated data, and inflates the number of samples by N+1 times by combining the data of the original sample with the N pieces of data obtained by the scaling processing. Alternatively, the learning model acquisition unit 43 can perform scaling processing at a magnification generated by a random number for each sample. The method of selecting the number of data and the method of setting the magnification in the inflating processing are not limited to the example described above, and the learning model acquisition unit 43 can perform inflating processing of the learning data using any other method.

[0066] Furthermore, if the coordinate transformation fixes the positions of the two feature points, the learning model acquisition unit 43 can perform the augmentation process by a coordinate transformation other than the scaling process in the x-axis direction. For example, after applying a nonlinear transformation to a sample (a pair of an image and contour information), it is possible to use the data that has been subjected to translation, rotation, and scaling processes so that the positions of the two feature points return to predetermined coordinates in the normalized space as the augmented data.

[0067] In this embodiment, the padding process when the learning data is normalized using two feature points has been described, but the learning model acquisition unit 43 can similarly perform padding process on the learning data when the learning data is normalized using a number of feature points other than two. For example, when one feature point (e.g., the apex position) is used for normalization, the learning model acquisition unit 43 can perform padding process by performing translation, rotation, and scale conversion so that the position of the one feature point does not change.

[0068] In addition, when three feature points (for example, the positions of the apex and the left and right valve annulus) are used for normalization, the learning model acquisition unit 43 can perform padding processing by calculating deformations such as expansion and contraction of the right ventricular region while keeping these three points fixed.

[0069] In addition, when four or more feature points are used for normalization, the learning model acquisition unit 43 can perform padding processing by fixing those four or more feature points as control points and randomly varying the weights of the transformation parameters.

[0070] In addition, the learning model acquisition unit 43 can impose restrictions so that the shape of the contour after the padding process does not become a statistically impossible shape. When the variation value exceeds a predetermined allowable range, the learning model acquisition unit 43 restricts the addition of the variation value. The learning model acquisition unit 43 sets the allowable range based on the average value and standard deviation of the learning data after the coordinate transformation into the normalized space. For example, the average value and standard deviation value (σ) of the parameter to be padded (scaling component in the x-axis direction when two points, the apex position and the midpoints of the left and right annulus, are used) are calculated from the learning data after the coordinate transformation into the normalized space, and padding process is performed within the range of the average ±2σ. By restricting the range of the learning data used for padding process in this way, it is possible to reduce the possibility that a statistically impossible shape is used as learning data by padding process, thereby reducing the quality (accuracy) of the learned model.

[0071] (Step S305: Building a trained model) In step S305, the learning model acquisition unit 43 performs statistical analysis on all of the samples of the learning data that have been augmented in step S304, and constructs a learned model. All of the processing in this step is performed in the normalized space.

[0072] As explained in step S303, the image handled in this embodiment is an image composed of Nx×Ny pixels. The pixel value information of the image in this embodiment is composed of a column vector in which the pixel values ​​of each pixel are arranged in the raster scan order of the image. That is, the upper left corner of the image shown in FIG. 4(b) is set as the origin (0,0), the pixel value at the location of pixel (x, y) is represented as I(x, y), and the pixel value information a of the image is defined as shown in the following formula 1.

[0073]

number

[0074] Moreover, the contour information in this embodiment is expressed by the coordinates of a predetermined number (Np) of points (contour point group). Each of these points is also expressed as (x, y) using the coordinate values ​​of the x-axis and y-axis, with the upper left corner of the image being the origin (0, 0). If the first point is p1, the coordinates of point p1 are (x1, y1). The contour information of the object in this embodiment is configured as a column vector with Np×2 elements that lists the x-coordinate values ​​and y-coordinate values ​​of each point placed on the contour of the object. The contour information b representing the right ventricle is defined as in the following formula 2.

[0075]

number

[0076] After acquiring the pixel value information a and the contour information b, the learning model acquisition unit 43 connects these column vectors to generate one column vector that combines the two pieces of information (pixel value information a, contour information b). That is, the learning model acquisition unit 43 generates one column vector for one sample that includes an element corresponding to the pixel value information a and an element corresponding to the contour information b of the object. That is, the learning model acquisition unit 43 generates data c as shown in the following formula 3.

[0077]

number

[0078] Here, since the pixel value information of the image and the contour information of the object have different variances, the learning model acquisition unit 43 may weight at least one of the pixel value information of the image and the contour information of the object. In this case, the learning model acquisition unit 43 may weight the pixel value information and the contour information according to the variances of the pixel value information and the contour information of the learning data so that the variances of the pixel value information and the contour information are equal or in a predetermined balance. Alternatively, the learning model acquisition unit 43 may weight the pixel value information and the contour information using a weight set by the user via the operation unit 35.

[0079] The learning model acquisition unit 43 generates data c by linking pixel value information and contour information for each sample (pair of image and contour information) included in the learning data (after padding) in the above-mentioned manner. Then, the learning model acquisition unit 43 performs statistical analysis on a data group consisting of data c of all samples included in the learning data to obtain a subspace. The learning model acquisition unit 43 can use a known method such as principal component analysis (PCA) as the statistical analysis, and can also use other methods such as kernel PCA and weighted PCA. The learning model acquisition unit 43 acquires a trained model by principal component analysis based on the PCA (Principle Component Analysis) method using an image of a predetermined object and the contour of the object as input data. The learning model acquisition unit 43 performs principal component analysis to calculate an average vector and eigenvectors for data c in which pixel value information and contour information are integrated, and further, eigenvalues ​​corresponding to each eigenvector. Here, by using the average vector (c bar) of data c concatenated with pixel value information and contour information, eigenvector e, and coefficient g for each eigenvector e, point d existing in the subspace related to the training data can be expressed by the following equation 4.

[0080]

number

[0081] Here, the c bar indicates the mean vector of data c, and e i is the eigenvector in the i-th basis, g i is a coefficient corresponding to the eigenvector. Furthermore, L represents the number of eigenvectors used in the calculation. A specific value of the number L of eigenvectors can be determined based on a cumulative contribution rate that can be calculated from the eigenvalues. For example, a cumulative contribution rate (e.g., 95%) is set in advance as a threshold, and the learning model acquisition unit 43 calculates the number of eigenvectors whose cumulative contribution rate is 95% or more, and sets this as L. Alternatively, the learning model acquisition unit 43 may set the number of eigenvectors set by the user as L.

[0082] As the final process in step S305, the learning model acquisition unit 43 stores the subspace information of the learning data obtained as a result of the above statistical analysis as a learned model in the RAM 33. Here, the subspace information is information that defines the subspace, and includes, for example, information on the eigenvectors and mean vectors that configure the subspace.

[0083] In step S203, the learning model acquisition process executed by the learning model acquisition unit 43 is a process independent of the contour estimation executed by this device (image processing device 10) for the input image. Therefore, the process of step S203 may be performed in advance, and the generated learning model may be stored in a storage device (for example, the database 22 or the storage unit 34). In this case, in step S203, the learning model acquisition unit 43 performs a process of reading the learning model generated in advance from the storage device and storing it in the RAM 33. By generating the learning model in advance, it is possible to reduce the processing time when performing the contour estimation process of the object in the input image. Note that the generation process of the learning model may be performed by a device other than the image processing device 10 according to the procedure shown in step S203.

[0084] In addition, in the present embodiment, an example is shown in which data in which pixel values ​​of the entire image are arranged in raster scan order is used as pixel value information, but other features related to pixel values ​​(features related to the texture of the image, etc.) may be used as pixel value information. Furthermore, in each sample of the learning data, data in which pixel values ​​of a partial image representing a part of the image are arranged may be used as pixel value information, or data in which only pixel values ​​around the contour of an object in the image are arranged may be used as pixel value information.

[0085] Alternatively, data (vectors) of principal component scores obtained by projecting an image onto a principal component space obtained by principal component analysis of sample images included in the training data may be used as pixel value information of the image.

[0086] For example, the learning model acquisition unit 43 calculates pixel value information a for each sample included in the learning data, and then performs principal component analysis on the data group of pixel value information a. The learning model acquisition unit 43 may use a vector of principal component scores of the image of each sample obtained by the principal component analysis as new pixel value information a' of the sample, instead of the pixel value information a. When the number of samples in the learning data is smaller than the number of pixels constituting the image, the number of dimensions of the pixel value information a' is smaller than the number of dimensions of the pixel value information a. Therefore, by using the vector of principal component scores of the image of each sample as new pixel value information a' of the sample, it is possible to reduce the calculation cost of the statistical analysis of the data in which the pixel value information and the contour information are linked.

[0087] In addition, by setting a threshold value for the cumulative contribution rate and reducing the number of dimensions of the principal component scores (i.e., the number of eigenvectors), it is possible to further reduce the calculation cost.

[0088] On the other hand, although the coordinate values ​​of the points (contour points) representing the contour of the object are used as the contour information of the object, other values ​​may be used. As an example, the contour information may be information obtained by arranging the results of calculating a level set function representing the object (for example, a signed distance value from the contour with the inside of the object being negative and the outside being positive) for each pixel of the image in raster scan order.

[0089] Alternatively, a label image or a mask image that distinguishes between the object and other objects may be used as the contour information. As described above, similar to the method of calculating pixel value information a' from pixel value information a, the learning model acquisition unit 43 may perform principal component analysis on the data group of contour information b, and use the value obtained by the principal component analysis as new contour information b' instead of the contour information b. That is, by performing principal component analysis on the data group of contour information b of samples included in the learning data, and setting the vector of the principal component score of the image of each sample as new contour information b' of the sample, it is possible to further reduce the calculation cost of the statistical analysis of the data in which the pixel value information and the contour information are linked. The above process ends the construction process of the learned model, and the process returns to step S204 in FIG. 2.

[0090] (Step S204: Transformation of coordinates into normalized space) In step S204, the contour estimation unit 44 acquires a normalized image by converting the input image into a normalized space so as to match the positions of the input points with the positions of the feature points of the trained model. That is, the contour estimation unit 44 converts the input image acquired by the image acquisition unit 41 in step S201 into the same space (i.e., the normalized space) as the normalized space in which the trained model was generated, using the information of the feature points acquired by the position acquisition unit 42 in step S202. Then, the contour estimation unit 44 calculates pixel value information of the input image in the normalized space.

[0091] Fig. 5 is a diagram showing the flow of the contour estimation process, and the process of this step will be specifically described using Fig. 5. Fig. 5(a) is a diagram showing the relationship between the input image and feature points (two points, the apex of the heart and the midpoints of the left and right annulus) before coordinate transformation into the normalized space. In this step, the contour estimation unit 44 performs coordinate transformation on the input image by translation, rotation, and scale transformation so that the coordinate values ​​of these feature points on the input image become the same as the coordinates of the feature points in the normalized space in which the trained model is defined.

[0092] When the number of feature points acquired by the position acquisition unit 42 is two, the contour estimation unit 44 acquires a normalized image in which the positions are aligned in the normalized space by rigid body transformation for translating and rotating the two feature points and by scale transformation for enlarging or reducing the distance between the feature points. For example, the position indicating the apex after the coordinate transformation (504 in FIG. 5(b)) has the same coordinates (300, 100) as the position indicating the apex in the learning data (401 in FIG. 4). Then, the position indicating the midpoint of the left and right valve annuli after the coordinate transformation (506 in FIG. 5(b)) has the same coordinates (300, 400) as the midpoint of the left and right valve annuli in the learning data (402 in FIG. 4). The method of coordinate transformation is the same as the method described above in step S303. Then, the contour estimation unit 44 crops the input image so that the number of pixels of the input image after the coordinate transformation becomes Nx×Ny.

[0093] In this embodiment, the normalization process of the input image has been described using an example in which the feature points are two points, the apex of the heart and the midpoints of the left and right annulus. However, the type of feature points, the number of feature points, and the normalization method using them need to be matched to the normalization process method adopted by the trained model. In either case, the coordinate conversion by the contour estimation unit 44 may be performed by using the same method as in step S303 to convert the coordinates of the input image into the normalized space. For example, when the number of feature points acquired by the position acquisition unit 42 is three, the contour estimation unit 44 performs affine transformation to acquire a normalized image in which the input image is converted into the normalized space. Alternatively, when the number of feature points acquired by the position acquisition unit 42 is four (including the case of four or more), the contour estimation unit 44 performs non-rigid transformation to match the positions of the acquired feature points by translating or rotating each feature point, and acquires a normalized image in which the input image is converted into the normalized space. Here, the contour estimation unit 44 can use a known method such as spline transformation or FFD (Free form deformation) as the non-rigid transformation.

[0094] (Step S205: Estimation of contour information) In step S205, the contour estimation unit 44 estimates contour information of an object appearing in the input image from the input image coordinate-converted into the normalized space in step S204 and the trained model acquired in step S203. More specifically, the contour estimation unit 44 estimates contour information of an object (right ventricular region) appearing in the image based on an estimation process (e.g., the BPLP method) that estimates information representing the posture of an object appearing in an unknown image from information on the pixel values ​​of the image, and stores the result in the RAM 33.

[0095] As an example of estimation processing, the BPLP method is a technique disclosed in, for example, the following non-patent document. The following non-patent document discloses a technique (BPLP method) for estimating information representing the posture of an object appearing in an image from pixel value information of an unknown image by utilizing subspace information related to data that combines pixel value information of a plurality of images and information representing the posture of the object appearing in the image. In other words, this technique is a technique for interpolating data in a missing portion from the results of statistical analysis of training data without any missing data, for data in which a certain missing portion has occurred. [Non-patent literature] Toshiyuki Amano, et.Al."An appearance based fast linear pose estimation" MVA 2009 IAPR Conference on Machine Vision Applications.2009 May 20-22. In order to apply the BPLP method, it is necessary to identify which parts of the input data are known information and which parts are unknown information (missing parts) when performing the estimation process.

[0096] In this embodiment, the contour estimation unit 44 sets pixel value information of the input image that has been coordinate-transformed into the normalized space as known information, and estimates the contour of the object in the input image by performing estimation processing based on the BPLP (Back Projection for Lost Pixels) method for data in which the contour of the object in the input image is set as unknown information. Specifically, the contour estimation unit 44 sets pixel value information of the input image that has been coordinate-transformed into the normalized space in step S204 as known information, and sets contour information of the object in the coordinate-transformed input image as unknown information. That is, the contour estimation unit 44 estimates the contour information by replacing the information settings in the above non-patent document so that "information of the pixel value of the image" is set to pixel value information a, and "information representing the posture of the object" is set to contour information b.

[0097] As a result of the estimation process, the contour estimation unit 44 calculates a vector f including elements corresponding to pixel value information a, which is known information, and elements corresponding to contour information b, which is unknown information, of the object, using the following equation 5.

[0098]

number

[0099] Here, f on the right-hand side of Equation 5 represents a column vector of input data in which a certain loss has occurred, and the column vector of f on the right-hand side is a column vector of data as shown in Equation 6 below, in which the portion of contour information b is set to 0 in the data that concatenates pixel value information a and contour information b as shown in Equation 3.

[0100]

number

[0101] In Expression 5, E is a matrix representing a subspace defined by the trained model acquired in step S203. When the L eigenvectors are e1, e2, . . . , eL, for example, the matrix E representing the subspace is given by E = [e1, e2, . . . , eL].

[0102] In Expression 5, Σ is a square matrix in which diagonal elements corresponding to pixel value information, which is known information, are set to 1, and other elements are set to 0. In other words, square matrix Σ is a matrix in which elements of a unit matrix corresponding to contour information, which is unknown information, are set to 0. In this embodiment, square matrix Σ is a square matrix with a side dimension of Nx×Ny+Np×2 (number of dimensions of pixel value information a+number of dimensions of contour information b), in which Np×2 diagonal elements corresponding to the number of dimensions of contour information b are 0, and the other diagonal elements are 1. Square matrix Σ is given in the format shown in Expression 7 below.

[0103]

number

[0104] In this embodiment, even if the input image is arbitrary, the portion set as unknown information does not change (it always becomes "0" corresponding to the portion of the contour information b). Therefore, in the learned model acquisition process in step S203, at the point in time when the learned model acquisition unit 43 calculates the subspace information, the contour estimation unit 44 can calculate in advance the matrix A of the following formula 8, which is a part of formula 5 to be obtained in the estimation process.

[0105]

number

[0106] Therefore, in the process of step S203 described above, the contour estimation unit 44 may calculate the matrix A in advance based on the formula 8, and the learning model acquisition unit 43 may use the calculation result as the learned model. In this case, in this step, the contour estimation unit 44 may calculate the vector f using the following formula 9.

[0107]

number

[0108] Note that the matrix A calculated in advance may be stored in a storage device (for example, the database 22 or the storage unit 34), and in the process of step S203, the learning model acquisition unit 43 may be configured to read the calculation result of the matrix A from the storage device as a learned model. This makes it possible to reduce the calculation cost related to the calculation for the estimation process. Note that a part of the matrix A (for example, (E T ΣE) -1 The calculation results of the above (part (a)) may be stored as a learned model in a storage device (e.g., database 22 or memory unit 34), and the learned model acquisition unit 43 may perform the remaining calculation processing in the learned model acquisition process, or the contour estimation unit 44 may perform the remaining calculation processing in the inference process.

[0109] Finally, the contour estimation unit 44 obtains information of the part corresponding to the contour information from the vector f indicating the result of the estimation process (in this embodiment, Np × 2 elements corresponding to the number of dimensions of the contour information b) and stores it in the RAM 33.

[0110] 5C is a diagram showing a schematic diagram of the estimation result of the contour line information. In the normalized space that has been subjected to the coordinate transformation in the processing in the previous step (step S204), contour line information 507 is estimated in the processing in this step.

[0111] If a vector of principal component scores of an image of each sample is used as new pixel value information a' of the sample instead of the pixel value information a when generating a trained model, the I(0,0) to I(Nx,Ny) parts of the column vector in Equation 6 may be replaced with values ​​calculated by the same method as the calculation method of the pixel value information a'. Specifically, when the pixel value information a' is the principal component score related to the pixel value information a of the training image, the contour estimation unit 44 may calculate the principal component score of the input image by performing a projection process onto the principal component space based on the input image and information on the subspace constructed using only the pixel value information a of the training data.

[0112] Furthermore, when generating a trained model, if a vector of principal component scores of an image of each sample is used as new contour information b' of the sample instead of the contour information b, the contour estimation unit 44 may estimate the principal component scores of the contour point coordinates in the input image using the above-mentioned method as unknown information. In this case, since the contour information of the estimation result is not in xy coordinates on the contour line, the contour estimation unit 44 may convert the estimated values ​​in the principal component space into coordinate values ​​and store them in the RAM 33.

[0113] Specifically, when the contour information b′ is a principal component score related to the contour information of the training data, the contour estimation unit 44 performs back projection processing based on the principal component score and subspace information constructed using only the contour information of the training data.

[0114] In addition, any calculation method may be used as long as it is a method for restoring missing information in principal component space, and is not limited to the BPLP method described above, and other calculation methods can be applied to the process of estimating contour information of an object.

[0115] (Step S206: Coordinate conversion to the original space) In step S206, the contour estimation unit 44 applies a transformation inverse to the coordinate transformation to the normalized space executed in step S204 to the result of contour estimation in step S205, thereby returning the result of contour estimation to the original (input image) space. By performing a coordinate transformation to return to the original space before the coordinate transformation, the contour information estimated in the normalized space in step S205 (507 in FIG. 5(c)) is coordinate-transformed to contour information in the original space (508 in FIG. 5(d)) as shown in FIG. 5(d).

[0116] (Step S207: Display of contour line estimation result) In step S207, the display processing unit 45 displays the input image and the contour information of the object estimated by the contour estimation unit 44 in the image display area of ​​the display unit 36. At this time, the display processing unit 45 can also perform display control so as to superimpose the estimated contour information and the input image. By superimposing the estimated contour information and the input image, the user can easily visually confirm how well the estimated contour information matches the input image. In this embodiment, the contour information of the object is a discrete point group obtained by sampling the contour of the object, so the display processing unit 45 may display the contour information on the display unit 36 ​​after interpolating between adjacent points using a known technique such as spline interpolation.

[0117] If the purpose is to analyze or measure an object, the process of step S207 is not necessarily required, and the configuration may simply involve storing the estimated contour information in a storage device.

[0118] In this embodiment, an example has been shown in which the coordinate values ​​of the contour point group representing the right ventricle are used as the contour information of the object, but the contour information may be a combination of coordinate values ​​of contour point groups representing two or more objects, such as the left ventricle, left atrium, right atrium, etc. in addition to the right ventricle. In this case, by performing a statistical analysis on not only the coordinate values ​​of the point group representing the right ventricle region but also the coordinate values ​​of all the point groups of the two or more objects in step S203, the coordinate values ​​of all the point groups of the two or more objects can be simultaneously estimated in step S205.

[0119] According to this embodiment, by using a trained model that has been subjected to statistical analysis in a normalized space that has been coordinate-transformed using the coordinates of specified feature points, it is possible to provide the user with highly accurate contour estimation results while keeping the feature points manually input by the user fixed in the normalized space.

[0120] (Modification) In the first embodiment, an example was shown in which the images treated as input images or learning data were two-dimensional ultrasound images of the cardiac region. However, the present invention can also be implemented in cases where images of organs other than the heart or images other than ultrasound images are used.

[0121] An example of application to images other than the heart or ultrasound images is the process of estimating the contour of the femur from a 2D X-ray image of the hip joint region. In the case of an image of a patient with osteoarthritis of the hip, the pelvis and femur are close to each other, and it may be impossible to calculate the boundary line by simple image processing such as threshold processing. In this case, the three points of the start, middle, and end positions of the junction of the femoral head and pelvis are manually input, and a trained model is constructed and the contour is estimated. In this way, it is possible to estimate the contour based on the statistical tendency of the femur shape by utilizing the characteristics of these input points.

[0122] In addition, in the above embodiment, an example was described in which one two-dimensional image (of a single time phase) is used, but the processing of this embodiment can also be applied to cases in which multiple two-dimensional images (of multiple time phases) are used.

[0123] For example, in step S201, an image of the end diastolic phase and an image of the end systolic phase of the heart are acquired as input images, and the processing of the above embodiment can be applied to the processing of estimating the contour of the heart cavity in each image.

[0124] In this case, for each sample of the training data, an image of the end diastolic phase and its correct contour information, an image of the end systolic phase and its correct contour information, and the coordinates of predetermined feature points on the image of any of the representative phases (e.g., the end diastolic phase) are prepared.

[0125] Then, in step S202, the position acquisition unit 42 acquires the coordinates of a predetermined feature point (input point) on the image of the representative phase of the input image. Then, in step S203, when generating a trained model, the trained model acquisition unit 43 performs coordinate conversion of the image of the representative phase into a normalized space using the coordinates of the feature points in the same manner as described above, and also performs coordinate conversion of the images of other phases into the same normalized space (using the coordinates of the feature points of the image of the representative phase). Then, the trained model acquisition unit 43 performs statistical analysis processing similar to that described above, using pixel value information a (Nx×Ny×2) obtained by arranging pixel values ​​of both images in raster scan order and contour point coordinates of both images arranged in order as contour information b (Np×2×2).

[0126] In step S204, the contour estimation unit 44 normalizes each image of the input image by a common coordinate transformation using information on the feature points (input points) of the image of the representative phase obtained in step S202. In addition, the contour estimation unit 44 executes the process of step S205 by expanding Equation 6 to the case where there are two input images. Note that the number of phases (number of images) used for estimation is not limited to two phases (two images), and can be expanded in the same manner even if it is any other number of phases (number of images). In addition, the contour estimation by the contour estimation unit 44 does not have to be performed for all phases, and for example, only the contour of the representative phase may be estimated. In this case, in step S203, only the contour point coordinates of the representative phase may be used as the above-mentioned contour information b. In addition, in step S205, the contour estimation unit 44 may estimate only the contour point coordinates of the representative phase as missing data.

[0127] In the above embodiment, the case where a two-dimensional image is used has been described as an example, but the processing of the above embodiment can also be applied to a case where an image having a dimension larger than a two-dimensional image, for example, a three-dimensional image, is used. For example, the processing of the above embodiment can be applied to a three-dimensional cardiac ultrasound image (3D-TTE image) captured using a transesophageal three-dimensional probe to similarly estimate the contour of the cardiac cavity. In this case, the size of the column vector representing the pixel value information a of the image in Equation 1 is Nx×Ny×Nz, where Nz is the pixel value in the Z direction. In addition, the contour is expressed by a mesh-shaped contour point group, and the number of elements of the vector representing the contour information in Equation 2 is configured as a column vector of the number of contour points Np×3.

[0128] In addition, in the case of performing contour estimation for a 3D-TTE image, it is possible to use other methods in addition to the above-mentioned method. For example, a predetermined cross section cut out from a three-dimensional image may be used as a reference cross section, and the process of the first embodiment may be applied to perform contour estimation of a two-dimensional image on the reference cross section, and then the contour line on the two-dimensional image may be used as an initial contour to calculate the contour of the region on the three-dimensional image. In this case, for example, the long axis of the right ventricular region is calculated from the initial contour, and the long axis is rotated by 180 degrees as the central axis to obtain an initial contour of the right ventricular region on the three-dimensional image. Then, the initial contour is deformed by a known method such as an Active-contour model or snakes to obtain a final contour. At this time, only the initial contour on the reference cross section (i.e., the cross section on which the user inputs the input point) is prevented from being displaced from the initial contour. In this way, it is possible to estimate the contour of the three-dimensional region while fixing it at the input point input by the user.

[0129] This makes it possible to perform highly accurate contour estimation that reflects the user's intentions.

[0130] <Second embodiment> The image processing device according to the second embodiment is a device that estimates the contour of a predetermined object from an image, similar to the first embodiment. In the first embodiment, a configuration for performing contour estimation using the BPLP method based on statistical analysis has been described, but in this embodiment, a configuration for performing contour estimation using machine learning represented by, for example, the CNN (Convolutional Neural Network) method will be described.

[0131] The device configuration and processing flowchart of the image processing device according to this embodiment are the same as those of the first embodiment. However, the construction process of the trained model (step S305) and the contour estimation process (step S205) are different. The differences from the first embodiment will be described below.

[0132] (Step S305: Building a trained model) In this embodiment, the learning model acquisition unit 43 constructs a learned model using VGG16, a known CNN model, as the basis of the CNN method. This VGG16 is a convolutional neural network (CNN method) consisting of a total of 16 layers. Since the number of channels of an input image is specified as 3 in VGG16, the learning model acquisition unit 43 stores the same image information in three channels.

[0133] In addition, in this embodiment, VGG16 is modified to suit the task of contour estimation. The original VGG16 is a network that performs image recognition of 1000 classes, and therefore the number of output parameters of the fully connected layer is 1000. This is changed to Np×2, which is the same as the number of elements to be estimated, that is, the number of elements in which the x coordinate value and the y coordinate value of each point arranged on the contour of the object are arranged as in the formula 2 in the first embodiment. The method of constructing the trained model can use the same method as the training method of the known CNN method. In this way, machine learning is performed using the CNN method, which inputs image information and outputs coordinate information of the contour line in the normalized space transformed by the coordinate conversion in step S304. The trained model acquisition unit 43 inputs an image (image information) of a predetermined object and acquires a trained model based on training data using the CNN method, which outputs contour information (position of the contour line) of the object in the captured image.

[0134] (Step S205: Estimation of contour information) In the method of estimating contour information, the contour estimation unit 44 can use the same method as the known estimation method of CNN. The contour estimation unit 44 inputs pixel value information of the input image that has been coordinate-transformed into the normalized space, and estimates the contour of the object in the input image based on learning by the CNN method that outputs the contour of the object in the input image. That is, the pixel value information of the input image that has been coordinate-transformed into the normalized space in step S204 is input, and the coordinate values ​​of the Np points constituting the contour of the object are estimated. In the estimation, the weight information of the CNN method learned in step S305 is used.

[0135] According to this embodiment, it is possible to estimate the contour of an object using the CNN method according to input points input by a user. This makes it possible to perform highly accurate contour estimation even when, for example, the contour of an object has a wide variety of variations and the expressive power of a trained model based on statistical analysis is insufficient.

[0136] In this embodiment, the CNN model is based on VGG16, but the CNN model is not limited to VGG16 and other models can be used. For example, a network in which the number of layers is reduced from VGG16 to such an extent that the convolution result covers the entire image may be used, or a new model may be constructed and machine learning may be performed. When the size of the network is reduced, for example, when the network is installed in an ultrasound diagnostic device, it is possible to perform contour estimation using the CNN method even if there are limitations on the computational resources.

[0137] <Third embodiment> The image processing device according to the third embodiment is a device that estimates the contour of a predetermined object from an image, similarly to the first and second embodiments. In this embodiment, a configuration is described in which, in each sample in the learning data, artificial variations are added to the coordinates of feature points used for normalization, thereby padding the learning data. In this embodiment, the learning model acquisition unit 43 functions as a normalization unit, and generates a normalized image and a normalized contour by a normalization process that converts the image and contour of the learning data into a normalized space so that new feature points to which variation values ​​have been added in the images of a plurality of cases are located at the same positions as new feature points to which variation values ​​have been added in the positions of the feature points. Then, the learning model acquisition unit 43 acquires a learned model based on the normalized image and the normalized contour.

[0138] The device configuration and processing flow chart of the image processing device according to this embodiment are the same as those of the first embodiment. However, the process of padding the learning data (step S304) is different. The differences from the first embodiment will be described below.

[0139] (Step S304: Inflation of learning data) In step S304, the learning model acquisition unit 43 performs a process of augmenting the learning data (data augmentation) in the normalized space subjected to the coordinate transformation in step S303.

[0140] First, the learning model acquisition unit 43 varies the coordinate values ​​of each feature point by a predetermined amount up, down, left, and right for each sample of the normalized learning data (by giving a variation value (Δx, Δy)), and records the coordinate values ​​(positions) after the variation. For example, the learning model acquisition unit 43 obtains a set of (N+1)×2 feature point coordinates, including the positions before the variation, by giving N variations to each of two feature points. Here, the variation value in the padding process of the learning data can be determined based on the variation of the coordinates of the feature points, which is obtained in advance by having a sufficient number of users manually input the coordinates of the feature points for a sufficient number of samples. For example, the variation of the coordinates of the manually input feature points can be modeled in advance by the standard deviation, and the variation value can be generated based on this model.

[0141] Also, a model of the variation in the coordinates of feature points may be set manually. In this case, the variation in the coordinates of feature points may be modeled for each type of feature point, or a model common to a plurality of feature points may be obtained. When the accuracy of the coordinates manually input by the user is high (when high accuracy can be expected), the variation value of the feature points can be set small, and when the accuracy of the coordinates manually input by the user is low (when high accuracy cannot be expected), the variation value of the feature points can be set large.

[0142] The learning model acquisition unit 43 adds different variation values ​​based on the positions of feature points that are allowed to vary. For example, the apex of the heart is a characteristic position on the image, and a small variation value is set therefor, since a high degree of accuracy is expected, whereas the midpoint of the left and right annulus does not have any particular characteristics on the image, and a high degree of accuracy is not expected, so a large variation value is set therefor.

[0143] Also, a model of the variation of the coordinates of the feature point may be determined based on the distribution of pixel values ​​around the feature point in each sample. That is, the learning model acquisition unit 43 sets the variation value based on the variance of pixel values ​​in the area around the feature point that allows variation. For example, in a small area around the feature point, the variance of pixel values ​​in each axial direction can be obtained, and a model can be set according to the variance. For example, a model can be set in which the variation is small if the variance is large, and a model can be set in which the variance is large if the variance is small. According to this, for feature points that have noticeable features such as lines and corners on the image, such as the apex of the heart, a model in the case of small variation (small variance) can be applied, and for feature points that do not have any particular features on the image, such as the midpoints of the left and right annuli, a model in the case of large variation (large variance) can be applied.

[0144] This model may be calculated for each sample in the normalized learning data, or a common model may be used by calculating a representative value based on multiple samples.Then, the fluctuation values ​​(Δx, Δy) may be generated by random numbers so that the distribution of the generated fluctuation values ​​follows the common model.

[0145] Alternatively, a reference value of the variation value for each feature point may be set according to the distance from each feature point to a contour point. That is, the farther the feature point is from the contour point, the larger the variation value may be set. Also, the number (N) for generating coordinates after variation may be a different value for each feature point. For example, the number (N) of feature points for which high accuracy can be expected may be set small, and the number (N) of feature points for which high accuracy cannot be expected may be set large.

[0146] The number of feature points (N) may be set manually, or may be set by calculating a value according to a variation model based on the above-mentioned variance of pixel values. For example, if the variation of pixel values ​​is small, a value that makes the number of feature points (N) small may be calculated and set, and if the variation of pixel values ​​is large, a value that makes the number of feature points (N) large may be calculated and set.

[0147] The variation value may be determined, for example, by taking into consideration the relative positional relationship between the position of the apex of the heart and the midpoint of the left and right annular positions in the learning data. For example, if the relative position of the midpoint of the left and right annular positions relative to the position of the apex of the heart in the vertical direction in all learning data is 100 pixels on average and the standard deviation σ is ±5 pixels, then the variation value may be ±10 pixels, which is twice the standard deviation (i.e., 2σ).

[0148] The variation value can be determined in the same manner for the left-right relative position of the midpoint of the left and right annulus positions relative to the apex position. The number (N) of feature points according to the variation value can be set arbitrarily, and in this example, it is assumed that the coordinates of a total of four feature points (i.e., N=4) after variation are obtained, one point each in the up-down direction and one point each in the left-right direction. Note that the method of setting the number (N) and the variation value for generating the coordinates after variation is not limited to the above example, and any other method can be used.

[0149] Secondly, the learning model acquisition unit 43 performs a coordinate transformation similar to the coordinate transformation into the normalized space performed in step S303 on each sample of the normalized learning data, with each of the (N+1)×2 sets of feature point coordinates as new feature point positions. As a result, the learning model acquisition unit 43 augments the learning data to which the positions (variation values) of the new feature points have been added by a factor of (N+1)×2.

[0150] It is to be noted that the training data can be augmented in a similar manner by performing the above-described feature point variation process and normalization process on training data samples before normalization, without performing the normalization process of step S303 on each sample of the training data after normalization.

[0151] Here, only the coordinate values ​​of the feature points are changed, and the image information of the learning data is not changed. By doing this, it is possible to represent in the learning data a situation in which, even if the image is the same, the coordinates of the feature points manually input for that image may vary slightly between multiple users or within the operations of each user.

[0152] Furthermore, even when the position of a point on the contour (contour point) and the position of a feature point overlap, such as the apex 401 illustrated in Fig. 4(a), i.e., even when the feature point and the contour point are the same point, the position of the correct contour point is not changed. By doing so, the position of the apex manually input by the user and the position of the apex as an estimated contour point do not necessarily match (although they always matched in the first embodiment). This makes it possible to prevent the position of the apex as an estimated contour point from being fixed at an incorrect position, which occurred in the first embodiment, when the position of the apex manually input by the user contains an error.

[0153] In addition, in this embodiment, the learning model acquisition unit 43 functioning as a normalization unit can restrict the addition of a variation value to a feature point that allows variation so that it does not become a statistically impossible value. When a variation value exceeds a predetermined allowable range, the learning model acquisition unit 43 restricts the addition of the variation value. The learning model acquisition unit 43 sets the allowable range based on the average value and standard deviation of the learning data after coordinate conversion to the normalized space. For example, the average value and standard deviation value (σ) of the relative position of a feature point (midpoint of the left and right annulus) that allows variation with respect to a fixed feature point (position of the apex of the heart) are calculated from the learning data after coordinate conversion to the normalized space, and a variation value is added within the range of the average ±2σ. By restricting the variation value to be added to a feature point that allows variation in this way, it is possible to reduce the possibility that a statistically impossible variation value is used, thereby reducing the quality (accuracy) of the learned model.

[0154] In the present embodiment, the case where the position of each feature point is changed has been described as an example. However, the feature points in each sample of the learning data may be classified into feature points whose positions are fixed and feature points whose positions are allowed to change in the normalized space, and only feature points whose positions are allowed to change may be changed. For example, a feature point having a high accuracy of an input point may be set as a fixed feature point. In this case, for example, the object is a cardiac chamber of the heart, and the apex of the cardiac chamber may be set as a fixed feature point. In addition, the position of the midpoint of the left and right annulus positions of the tricuspid valve related to the cardiac chamber may be set as a feature point whose changes are allowed. In addition, when a feature point and a contour point are the same point, that is, a feature point whose position overlaps with a contour point may be set as a fixed feature point. In addition, a configuration may be adopted in which a user can select a feature point to be fixed when generating a trained model.

[0155] In addition, in this embodiment, the case where the number of feature points is two has been described as an example, but the processing of this step can also be applied to cases where the number of feature points is one or three or more. In this case, it is possible to construct a trained model that flexibly meets the needs for the number of input points input by the user and the selection between fixed feature points and feature points that allow variation.

[0156] According to this embodiment, the coordinate values ​​of feature points that are allowed to vary are not necessarily fixed by the coordinate values ​​of the points input by the user, but may be changed to reflect the variation values ​​added when constructing the trained model. Therefore, it is possible to estimate contour information without strict position specification when inputting the input points, which improves convenience for the user.

[0157] On the other hand, according to the first embodiment, the feature point positions input by the user are always fixed, which has the effect of obtaining contour estimation results that match the user's intention (for example, for a user who is confident in the input points). Since there is a trade-off between these two, a trained model according to the first embodiment and a trained model according to this embodiment may be generated, and the trained model may be selected according to the user's preference or various case applications.

[0158] The process of padding the learning data described in this embodiment can be applied to both the case where the BPLP method is used as in the first embodiment and the case where the CNN method is used as in the second embodiment.

[0159] <Fourth embodiment> The image processing device according to the fourth embodiment is a device that estimates the contour of a predetermined object from an image, similarly to the first to third embodiments. In the first to third embodiments, a configuration has been described in which, when estimating the contour, input points manually input by a user are acquired, and the contour of the object is estimated by semi-automatic estimation using the input points. Here, the contour estimation of the object is not limited to processing using the input points, and it is also possible to adaptively switch between fully automatic contour estimation and semi-automatic contour estimation that do not depend on the input points depending on the situation.

[0160] The multiple learned models described in this embodiment include a first learned model that does not use user input points in contour estimation, and a second learned model that uses the input points in contour estimation.In this embodiment, contour estimation that does not use user input points in contour estimation is referred to as fully automatic contour estimation, and contour estimation that uses the input points in contour estimation is referred to as semi-automatic contour estimation.

[0161] In this embodiment, a configuration for performing contour estimation by adaptively switching between full-automatic contour estimation and semi-automatic contour estimation depending on the situation will be described. When performing contour estimation, first, full-automatic contour estimation is performed, and if the user determines that semi-automatic contour estimation is necessary, an example will be described in which input of an input point is accepted and semi-automatic contour estimation is performed. A case in which it is determined that semi-automatic contour estimation is necessary is when coordinate information (feature point coordinates) of a feature point (input point) is acquired in step S608 in Fig. 6 described below.

[0162] The device configuration of the image processing device according to this embodiment is similar to the device configuration (FIG. 1) of the first to third embodiments. However, the functions of the learning model acquisition unit 43 and the contour estimation unit 44 are different from those of the first to third embodiments.

[0163] The learning model acquisition unit 43 acquires learning data from the database 22 and acquires a learned model by performing statistical analysis on the learning data. The learning model acquisition unit 43 acquires a plurality of learned models that have learned the relationship between an image of a predetermined object and the contour of the object in the image.

[0164] In the first to third embodiments, the trained model is defined in a normalized space in which coordinates are transformed using coordinate information of feature points so that the coordinates of feature points in each sample are the same. That is, only a trained model for semi-automatic contour estimation is acquired, on the premise that the coordinates of feature points (input points) are input together with an image during contour estimation.

[0165] On the other hand, the learning model acquisition unit 43 according to this embodiment acquires multiple learned models generated in normalized spaces with different combinations of feature points (for example, combinations of positions and numbers in an object). That is, in this embodiment, not only a learned model for semi-automatic contour estimation, but also a learned model for fully automatic contour estimation that does not require input of coordinates of feature points is acquired. In this way, the learning model acquisition unit 43 in this embodiment acquires two types of learned models, a learned model for semi-automatic contour estimation and a learned model for fully automatic contour estimation.

[0166] The contour estimation unit 44 estimates the contour of the object in the input image using a learned model selected from a plurality of learned models (changing the type of learned model) depending on the presence or absence of an input point input by the user, and a normalized image coordinate-transformed into the normalized space of the learned model. When no input point is input, the contour estimation unit 44 estimates the contour of the object in the input image using a first learned model that does not use the input point by the user in contour estimation, and when an input point is input, the contour estimation unit 44 estimates the contour of the object in the input image using a second learned model that uses the input point in contour estimation.

[0167] When performing contour estimation using a trained model for fully automatic contour estimation, the contour estimation unit 44 obtains a normalized image by performing coordinate conversion of an input image into the same normalized space as the normalized space in which the trained model for fully automatic contour estimation was generated. Then, the contour estimation unit 44 estimates the contour of an object in the input image using the selected trained model and the normalized image.

[0168] Furthermore, when performing contour estimation using a trained model for semi-automatic contour estimation, the contour estimation unit 44 selects a trained model corresponding to a combination of input points (input points input in S608) from among multiple trained models (multiple trained models generated in S602), and obtains a normalized image by performing coordinate conversion of the input image into the same normalized space as the normalized space of the selected trained model. Then, the contour estimation unit 44 estimates the contour of the object in the input image using the selected trained model and the normalized image.

[0169] As in the first to third embodiments, the contour estimation unit 44 estimates the contour of the object in the input image acquired by the image acquisition unit 41 (for example, the position of the contour line as contour information) by matrix calculation using the trained model acquired by the learning model acquisition unit 43. While only semi-automatic contour estimation is performed in the first to third embodiments, in this embodiment, fully automatic contour estimation and semi-automatic contour estimation are switched according to the situation depending on the presence or absence of an input point. That is, in a situation where it is determined that fully automatic contour estimation should be performed, the contour estimation unit 44 estimates the contour of the object in the input image using the trained model for fully automatic contour estimation, with only the input image acquired by the image acquisition unit 41 as input (without using the coordinates of feature points).

[0170] On the other hand, in a situation where it is determined that semi-automatic contour estimation should be performed, the contour estimation unit 44 uses a learned model for semi-automatic contour estimation to estimate the contour of the object in the input image, using the input image acquired by the image acquisition unit 41 and the coordinates of feature points acquired by the position acquisition unit 42.

[0171] Next, an example of the processing procedure of the image processing device 10 in Fig. 1 will be described with reference to the flowchart in Fig. 6. Here, since steps S601, S605, and S606 in Fig. 6 are the same processes as steps S201, S206, and S207 in Fig. 2, respectively, their explanations will be omitted, and only differences from the process in Fig. 2 will be described below.

[0172] (Step S602: Obtaining a trained model) In step S602, the learning model acquisition unit 43 constructs a learned model for fully automatic contour estimation and a learned model for semi-automatic contour estimation from the learning data acquired from the database 22. Fig. 7 is a flowchart showing an example of the procedure of a learned model acquisition process according to the fourth embodiment, and the details of the process performed by the learning model acquisition unit 43 will be described below with reference to the flowchart in Fig. 7.

[0173] Here, the processes of steps S711 and S721 to S724 in Fig. 7 are the same as the processes of steps S301 to S305 in Fig. 3. However, this embodiment differs from the first embodiment in that a plurality of trained models having different types and numbers of feature points are obtained.

[0174] (S712: Coordinate transformation to normalized space (for fully automatic contour estimation)) In step S712, prior to constructing a learned model in the next step, the learning model acquisition unit 43 performs coordinate transformation of image information such as pixel value information of each sample image of the learning data into a normalized space that defines a learned model for fully automatic contour estimation. At this time, the learning model acquisition unit 43 performs coordinate transformation so that the coordinate values ​​of the probe position (e.g., 509 in FIG. 5(a)) in each sample image of the learning data match each other. That is, the learning model acquisition unit 43 performs coordinate transformation by translating the image of each sample of the learning data so that a predetermined position (e.g., probe position) in the image has a predetermined coordinate value.

[0175] In this embodiment, the coordinate transformation to the normalized space for fully automatic contour estimation is performed using the coordinate information (position information) of one point, the probe position, as an example. However, the coordinate transformation to the normalized space may be performed using other points, not limited to the probe position, as long as the position can be identified only from image information such as pixel value information in the image of each sample. In addition, the point (position) serving as the reference for the coordinate transformation is not limited to one point, and multiple points (positions) can be used as the reference for the coordinate transformation. For example, two points, the probe position and the septum center point, may be identified from the image information and the coordinate transformation may be performed. In this case, as in step S303 in the first embodiment, a coordinate transformation matrix is ​​calculated and applied by translation, rotation, and scale transformation so that these two points match. In addition, three or more points may be automatically detected using a known method, and coordinate transformation may be performed as in step S303.

[0176] (S713: Inflation of learning data (for fully automatic contour estimation)) In step S713, the learning model acquisition unit 43 performs data augmentation of the learning data in the normalized space obtained by the coordinate transformation in step S712. The learning model acquisition unit 43 can perform augmentation of each sample image by applying a known general method. For example, the learning model acquisition unit 43 executes a number of operations, such as translation, rotation, enlargement, or reduction, on the pixel value information of each sample image. The learning model acquisition unit 43 can also perform augmentation by making changes to pixel values, such as converting brightness contrast.

[0177] (S714: Building a trained model (for fully automatic contour estimation)) In step S714, the learning model acquisition unit 43 statistically analyzes all of the learning data acquired in step S711 and coordinate-converted in step S712, and the learning data augmented in step S713, to construct a learned model for fully automatic contour estimation.

[0178] The normalized space generated by the processing of this step is not a normalized space that has been coordinate-transformed so that the coordinates of feature points such as the apex of the heart and the midpoints of the left and right annulus positions match, but a normalized space that has been coordinate-transformed by translation, rotation, scale transformation, etc. so that the positions of each sample in the image (e.g., probe position) match.

[0179] The process in this step differs from that in the first embodiment in the process of generating the normalized space as described above, but the other processes in this step are the same as those in step S305 in the first embodiment. The above process completes the process of constructing a trained model for fully automatic extraction.

[0180] The processes in the following steps S721 to S724 are processes for constructing a trained model for semi-automatic estimation, and are similar to steps S302 to S305 in Fig. 3. However, the processes in steps S721 to S724 differ from the processes in the first embodiment in that the trained model corresponding to the input points is not limited, and the trained model acquisition unit 43 executes the processes in steps S721 to S724 to acquire multiple trained models with different types and numbers of feature points. The above processes end the process of constructing a trained model for semi-automatic extraction, and the process returns to step S603 in Fig. 6.

[0181] (Step S603: Transformation of coordinates into normalized space) In step S603, the contour estimation unit 44 functions as a selection unit and selects a trained model from multiple trained models (S602) depending on the presence or absence of an input point input by the user. The contour estimation unit 44 also functions as a normalization unit and acquires a normalized image by performing coordinate transformation on the input image acquired in step S601 into the same normalized space as the normalized space in which the selected trained model was generated. Then, the contour estimation unit 44 calculates pixel value information of the input image in the normalized space.

[0182] In contour estimation using a trained model for fully automatic contour estimation, coordinate transformation using input points cannot be performed in normalized space, so coordinate transformation to normalized space is performed using the same process as in step S712. That is, if coordinate transformation is performed in step S712 so that known positions (e.g., probe positions) in the images of each sample coincide with each other, the probe position is used in the same way. The contour estimation unit 44 estimates the contour of the object in the input image using the selected trained model and the normalized image.

[0183] On the other hand, in contour estimation using a trained model for semi-automatic contour estimation, coordinate transformation using an input point can be performed in a normalized space. When an input point is input, the contour estimation unit 44 acquires a normalized image by performing coordinate transformation of the input image into the normalized space so as to match the position of the input point with the position of a feature point of the trained model (trained model for semi-automatic contour estimation). Here, contour estimation using a trained model for semi-automatic contour estimation is similar to the processing described in the first to third embodiments.

[0184] The contour estimation unit 44 changes the process of this step depending on whether or not feature point coordinates have been acquired. When the coordinate information (feature point coordinates) of the feature point (input point) has not been acquired in step S608 of Fig. 6, and the processes of this step are executed in the order of steps S601, S602, and S603, the contour estimation unit 44 performs coordinate conversion of the input image into the same normalized space as the normalized space in which the trained model for fully automatic contour estimation was generated. The coordinate conversion process in this case is the same as the process of step S712 of Fig. 7.

[0185] On the other hand, when the coordinate information (feature point coordinates) of the feature point (input point) is acquired in step S608, that is, when the processing of this step is executed via step S608, the contour estimation unit 44 performs coordinate conversion of the input image into the same normalized space as the normalized space in which the trained model for semi-automatic contour estimation was generated. The coordinate conversion processing in this case is the same as the processing in step S722 (S303 in FIG. 3).

[0186] As described above, after the contour estimation unit 44 performs coordinate conversion on the input image by processing in step S712 or S722 (S303 in FIG. 3), the contour estimation unit 44 crops the input image so that the number of pixels of the input image after coordinate conversion becomes Nx × Ny, as in step S204.

[0187] (Step S604: Estimation of contour information) In step S604, the contour estimation unit 44 estimates contour information of an object appearing in the input image from the input image coordinate-converted in step S603 and the trained model acquired in step S602. More specifically, the contour estimation unit 44 estimates contour information of an object appearing in the image (right ventricular region) based on the BPLP method, and stores the result in the RAM 33.

[0188] The contour estimation unit 44 selects (changes) the learned model to be used for contour information estimation depending on whether or not coordinate information (feature point coordinates) of the feature point (input point) has been acquired in step S608 of Fig. 6. That is, if coordinate information (feature point coordinates) of the feature point (input point) has not been acquired in step S608 of Fig. 6, the contour estimation unit 44 selects a learned model (S714) for fully automatic contour estimation in contour information estimation. On the other hand, if coordinate information (feature point coordinates) of the feature point (input point) has been acquired in step S608 (via step S608), the contour estimation unit 44 selects a learned model (S724) for semi-automatic contour estimation in contour information estimation.

[0189] At this time, the trained model for semi-automatic contour estimation is used according to the combination (type and number) of feature points acquired in step S608. For example, when the apex position is input (acquired) as a feature point, the trained model used is a trained model normalized with one point of the apex position. Note that, except for switching the trained model in this way, the process of this step is the same as the process of step S205 in the first embodiment.

[0190] (Step S607: Obtain an instruction as to whether or not to end contour estimation) In step S607, the display processing unit 45 displays a confirmation screen in the image display area of ​​the display unit 36 ​​for acquiring an instruction from the user as to whether or not to terminate the contour line estimation. For example, the display processing unit 45 displays a dialog box having two buttons, "End" and "Do not end", on the display unit 36 ​​as an instruction input unit for inputting an instruction from the user. In addition, any method can be used for acquiring an instruction from the user, such as accepting a predetermined keyboard input. If an instruction to "end" the contour line estimation process is input (S607-Yes), the contour line estimation process by the image processing device 10 of this embodiment is terminated. On the other hand, if an instruction to "not end" the contour line estimation process is input in the determination process of step S607 (S607-No), the process proceeds to step S608.

[0191] (Step S608: Obtaining feature point coordinates) In step S608, the position acquisition unit 42 accepts an input of coordinate information of a feature point (input point) by the user. The position acquisition unit 42 acquires the coordinate information of a predetermined feature point, and stores it in the RAM 33.

[0192] In step S608, the display processing unit 45 causes the display unit 36 ​​to display a superimposed image obtained by superimposing the result of the estimation in step S606 on the input image. Then, the position acquisition unit 42 acquires the position of the feature point input by the user on the superimposed image as an input point.

[0193] The feature point input by the user is determined by which trained model for semi-automatic contour estimation has been acquired by the process of step S602. For example, if no feature point has been input yet, that is, if this step (S608) is executed for the first time, the feature point input in this step will be the "first" feature. Then, if a trained model for semi-automatic contour estimation with one feature point is normalized at the apex position, this trained model is selected, and the "predetermined feature point" input by the user is determined as the apex position.

[0194] Next, if this step (S608) is executed with one feature point already input at the apex position, the feature point input in this step will be the "second" feature. If a trained model for semi-automatic contour estimation with two feature points is normalized at two points, the apex position and the midpoint of the left and right valve annulus positions, this trained model is selected, and the second "predetermined feature point" input by the user is determined to be the midpoint of the left and right valve annulus positions.

[0195] In this way, each time this step (S608) is executed, one feature point is added, and in the subsequent steps, a trained model for semi-automatic contour estimation that corresponds to the one or more input feature points is selected, and semi-automatic contour estimation is performed based on the selected trained model for semi-automatic contour estimation.

[0196] In step S608 in Fig. 6, the input of feature points may be added one by one, or multiple feature points may be added together. In this case, the trained model for semi-automatic contour estimation selected corresponds to the number of input feature points. For example, when multiple feature points (N points) are input, the trained model for semi-automatic contour estimation corresponding to the N points is selected, and semi-automatic contour estimation is performed based on the selected trained model.

[0197] If a trained model for semi-automatic contour estimation corresponding to the number of input feature points has not been acquired in the process of step S602, the contour estimation unit 44 does not perform contour estimation using the trained model, and outputs a curve obtained by interpolating the input feature points as a contour. For example, when three types of trained models for semi-automatic contour estimation have been acquired corresponding to three types of feature points (one point, two points, three points) in the process of step S602, if a fourth feature point (N=the fourth feature point) is input, the contour estimation unit 44 outputs a curve obtained by smoothly interpolating these four points by spline interpolation as a contour.

[0198] In this embodiment, an example has been described in which fully automatic contour estimation is performed first, and then semi-automatic contour estimation is performed if the user determines that it is necessary. However, it is also possible to perform processing to skip fully automatic contour estimation, for example, based on the evaluation results of the image quality of the input image input in step S601.

[0199] In this case, the image acquisition unit 41 acquires an input image and evaluates the image quality of the input image. Then, the contour estimation unit 44 functions as a selection unit and selects a trained model from a plurality of trained models based on the image quality evaluation result. The contour estimation unit 44 also acquires a normalized image by transforming the input image acquired in step S601 into the same normalized space as the normalized space in which the selected trained model was generated. Then, the contour estimation unit 44 estimates the contour of the object in the input image using the selected trained model and the normalized image.

[0200] When the image quality of the input image is evaluated as not satisfying the threshold, the contour estimation unit 44 estimates the contour of the object in the input image by using a second trained model that uses the input points in contour estimation, rather than using a first trained model that does not use the input points in contour estimation.

[0201] When the image acquisition unit 41 (evaluation unit) evaluates that the image quality of the input image does not satisfy the threshold, the display processing unit 45 displays a user interface that prompts the input of feature points on the display unit 36, and the position acquisition unit 42 acquires the positions of the feature points input from the user interface as input points. The contour estimation unit 44 uses the coordinates of the inputted predetermined feature points to perform semi-automatic contour estimation using the second trained model.

[0202] As an example of image quality evaluation, the image acquisition unit 41 (evaluation unit) acquires an input image in step S601, then applies a Fourier transform to the input image to convert it into the frequency domain, and evaluates that the image quality of the input image does not satisfy the noise threshold if the high-frequency components are equal to or greater than a certain level. Alternatively, the image acquisition unit 41 (evaluation unit) calculates the contrast of the input image, and evaluates that the image quality of the input image does not satisfy the contrast threshold if the contrast is equal to or less than a certain contrast threshold. If it is evaluated that the image quality of the input image does not satisfy a threshold (for example, a noise threshold or a contrast threshold), the full-automatic contour estimation is skipped, and the contour estimation unit 44 performs semi-automatic contour estimation.

[0203] (Modification) In the processing of Figures 6 and 7, multiple trained models are acquired in step S602, and a trained model for semi-automatic contour estimation corresponding to the feature points acquired in step S608 is selected to perform semi-automatic contour estimation.

[0204] Note that the process of acquiring a trained model is not limited to this example. For example, as shown in FIG. 8, depending on the presence or absence of feature points (input points), it is also possible to divide the process into a process of constructing a trained model for fully automatic estimation (S711 to S714) and a process of constructing a trained model for semi-automatic estimation (S721 to S724).

[0205] 8 is a flowchart showing a modified example of the trained model acquisition process according to the fourth embodiment. In this flowchart, the process corresponding to step S608 (FIG. 6) is executed before the trained model acquisition process. That is, before the trained model acquisition unit 43 generates multiple trained models, the presence or absence of feature points is determined in step S800.

[0206] For example, if a feature point (input point) has not been input and coordinate information of the feature point (input point) has not been acquired (S800-No), the learning model acquisition unit 43 advances the process to S711 and executes the process of constructing a learned model for fully automatic contour estimation (S711 to S714).

[0207] On the other hand, if it is determined in step S800 that feature points (input points) have been input and coordinate information of the feature points (input points) has been acquired (S800-Yes), the learning model acquisition unit 43 advances the process to S721 and executes the process of constructing a learned model for semi-automatic contour estimation (S721 to S724) corresponding to the combination of acquired feature points (combination of positions and numbers in the target object).

[0208] According to this embodiment, first, a fully automatic contour estimation result is displayed, and then a trained model is switched in response to interactive input of feature point coordinates, thereby making it possible to perform semi-automatic contour estimation based on the input feature points. This makes it possible to perform highly accurate contour estimation that reflects the user's intention.

[0209] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0210] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0211] 10 Image processing device 21 Network 22 Database 31 Communication Interface 32 ROM 33 RAM 34 Storage section 35 Control section 36 Display section 37 Control Unit 41 Image acquisition unit 42 Position acquisition part 43 Learning model acquisition unit 44 Contour Estimation Unit 45 Display processing section

Claims

1. A learning device for acquiring a trained model in which a contour of an object in an image of the object is trained based on positions of predetermined feature points, a learning data acquisition unit that acquires, as learning data, positions of contours of objects in images of a plurality of cases and positions of feature points of the objects in the images; a normalization unit that generates normalized images and normalized contours by a normalization process that converts the images and the contours into a normalized space so that the feature points in the images of the plurality of cases are at the same positions; a deformation unit that generates a deformed image and a deformed contour by applying deformation to the normalized image and the normalized contour while fixing the positions of the feature points at the same positions in the normalized space; a model acquisition unit that acquires the trained model based on the deformed image and the deformed contour; Equipped with The learning device is characterized in that the deformation unit generates the deformed image and the deformed contour by adding a variation value that translates a position in the normalized space, and limits the addition of the variation value if the variation value exceeds a predetermined tolerance range.

2. The learning device according to claim 1 , wherein the deformation unit generates the deformed image and the deformed contour by adding a fluctuation value for rotating the image in the normalized space.

3. The learning device according to claim 1 or 2, characterized in that the deformation unit generates the deformed image and the deformed contour by adding a positional fluctuation value obtained by enlarging or reducing a relative distance from a position of the fixed feature point.

4. 4. The learning device according to claim 1, wherein the transformation unit sets the permissible range based on a mean value and a standard deviation of the learning data after coordinate transformation into the normalized space.

5. A learning device for acquiring a trained model in which a contour of an object in an image of the object is trained based on positions of predetermined feature points, a learning data acquisition unit that acquires, as learning data, positions of contours of objects in images of a plurality of cases and positions of feature points of the objects in the images; a normalization unit that generates a normalized image and a normalized contour by a normalization process that converts the image and the contour into a normalized space in coordinates so that new feature points to which a variation value has been added at a position of the feature point are positioned in the same position in the images of the plurality of cases; A model acquisition unit that acquires the trained model based on the normalized image and the normalized contour; Equipped with the normalization unit classifies the feature points in the normalized space into fixed feature points whose positions are fixed, and variation-tolerant feature points whose positions are allowed to vary by adding the variation value.

6. The learning device according to claim 5 , wherein the normalization unit adds a different variation value based on the position of the feature point for which variation is permitted.

7. The learning device according to claim 5 , wherein the normalization unit sets the variation value based on a variance value of pixel values ​​in a region surrounding the feature point for which variation is permitted.

8. 8. The learning device according to claim 5, wherein the object is a cardiac chamber, and the fixed feature point is an apex position of the cardiac chamber.

9. 9. The learning device according to claim 5, wherein the object is a cardiac chamber, and the feature point that allows the fluctuation is a position of the midpoint between the left and right annulus positions of the tricuspid valve associated with the cardiac chamber.

10. A learning device according to any one of claims 1 to 9; an image acquisition unit for acquiring an input image; a position acquisition unit that acquires a position of an input point input by a user on the input image; a normalization unit that acquires a normalized image by performing coordinate transformation on the input image so as to align a position of the input point with a position of a predetermined feature point in a trained model acquired by the learning device; an estimation unit that estimates a contour of the object in the input image by using the normalized image and the trained model; An image processing device comprising:

11. A medical image capturing apparatus having an image processing device for processing captured images, The image processing device includes: A learning device according to any one of claims 1 to 9; an image acquisition unit for acquiring an input image; a position acquisition unit that acquires a position of an input point input by a user on the input image; a normalization unit that acquires a normalized image by performing coordinate transformation on the input image so as to align a position of the input point with a position of a predetermined feature point in a trained model acquired by the learning device; an estimation unit that estimates a contour of the object in the input image by using the normalized image and the trained model; A medical imaging apparatus comprising:

12. A learning method for a learning device that acquires a trained model in which a contour of a predetermined object in an image of the object is trained based on positions of predetermined feature points, comprising: a learning data acquiring step of acquiring, as learning data, positions of contours of objects in images of a plurality of cases and positions of feature points of the objects in the images; a normalization step of generating normalized images and normalized contours by a normalization process of converting the images and the contour positions into a normalized space so that the feature points in the images of the plurality of cases are at the same positions; a deformation step of generating a deformed image and a deformed contour by applying deformation to the normalized image and the normalized contour while fixing the positions of the feature points in the normalized space; A model acquisition step of acquiring the trained model based on the deformed image and the deformed contour; having The learning method is characterized in that the transformation process generates the deformed image and the deformed contour by adding a variation value that translates a position in the normalized space, and limits the addition of the variation value if the variation value exceeds a predetermined tolerance range.

13. The learning method described in Claim 12, characterized in that the deformation process generates the deformed image and the deformed contour by adding a fluctuation value that rotates in the normalized space.

14. The learning method described in Claim 12 or 13, characterized in that the deformation process generates the deformed image and the deformed contour by adding a position fluctuation value that expands or contracts the relative distance to the position of the fixed feature point.

15. A learning method for acquiring a trained model in which a contour of an object in an image of the object is trained based on positions of predetermined feature points, comprising: a learning data acquiring step of acquiring, as learning data, positions of contours of objects in images of a plurality of cases and positions of feature points of the objects in the images; a normalization step of generating a normalized image and a normalized contour by a normalization process of converting the image and the contour into a normalized space in coordinates so that the new feature points to which the variation values ​​have been added are positioned in the same positions as the new feature points to which the variation values ​​have been added in the images of the plurality of cases; A model acquisition step of acquiring the trained model based on the normalized image and the normalized contour, the normalization step classifies the feature points in the normalized space into fixed feature points whose positions are fixed, and variation-tolerant feature points whose positions are allowed to vary by adding the variation value.

16. A program for causing a computer to execute each step of the learning method according to any one of claims 12 to 15.

Images