Image processing apparatus, image processing method, and program

Abstract

To provide a technology for accurately extracting information on a shape of an area of interest from a medical image.SOLUTION: An image processing apparatus for estimating shape information, which is information on a shape of an area of interest, from an image includes: image acquisition means which acquires a first image to be processed; partial space information acquisition means which acquires two or more pieces of partial space information generated with different datasets; partial space information selection means which selects first partial space information from the two or more pieces of partial space information, on the basis of pixel value information of the first image; and estimation means which estimates shape information of an area of interest in the first image from the pixel value information of the first image, by use of the first partial space information. The dataset includes sample data including pixel value information of a learning image and the shape information of the area of interest.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In the medical field, diagnoses are made using images acquired by various imaging devices (modalities) such as ultrasound diagnostic imaging devices. In these diagnoses, information such as the area, volume, and dimensions of a region of interest shown in an image is used for diagnosis. In order to calculate the area of ​​the region, it is necessary to extract the contour of the region from the image (to estimate contour information, which is information that represents the contour shape). However, when this region extraction work is performed manually, it is an issue that the operator is forced to work very hard. For this reason, various techniques have long been proposed for automatic or semi-automatic region extraction from images in order to reduce the operator's work.

[0003] As an example, as in the following prior art, there is a method of using images and correct contour information of a region of interest in the image collected for many cases as learning data. In this technology, contour information of a region of interest in an unknown input image is estimated (i.e., contour is extracted) based on the result of performing statistical analysis on the learning data. Non-Patent Document 1 discloses a technology for constructing a statistical model called an Active Appearance Model based on information on pixel values ​​of an image of learning data and information on coordinate values ​​of a point group representing the contour of a region of interest in the image. Furthermore, a technology for estimating contour information of a region of interest in the input image by using an iterative process based on a gradient method, utilizing an evaluation value representing the similarity between pixel value information of an unknown input image and pixel value information of an image in the statistical model, is also disclosed. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Elco Oost,et.al.”Active Appearance Models in Medical Image Processing” Medical Imaging Technology.vol.27.No3.2009 May. Summary of the Invention [Problem to be solved by the invention]

[0005] In estimation processing using a statistical model, a plausible estimation result is obtained when an image close to the sample data group included in the training data is input, but in rare cases, an inappropriate estimation result may be output when an image far from the sample data group is input. In general, when generating a statistical model, it is considered desirable to have as much variation in the training data as possible. This is because it is expected that the more variation in the training data increases, the higher the generalization ability of the statistical model will be, and the more unknown data will be able to be handled. However, the more the variation in the training data is increased and the generalization ability of the statistical model is improved, the more likely it is that the inappropriate estimation result described above will be output.

[0006] The present invention has been made in view of the above-mentioned problems, and has an object to provide a technique that can extract information relating to the shape of a region of interest from a medical image with higher accuracy.

[0007] In addition to the above object, the present invention also provides effects and advantages that cannot be obtained by conventional techniques, which are derived from the configurations shown in the embodiments of the invention described below. This can be seen as one of the other purposes of the disclosure of the book. [Means for solving the problem]

[0008] A first aspect of the present invention provides an image processing device that estimates shape information, which is information regarding the shape of a region of interest from an image, comprising: an image acquisition means for acquiring a first image to be processed; a subspace information acquisition means for acquiring information of two or more subspaces generated using different datasets; a subspace information selection means for selecting information of a first subspace from the information of the two or more subspaces based on pixel value information of the first image; and an estimation means for estimating shape information of a region of interest in the first image from pixel value information of the first image using the information of the first subspace, wherein the dataset includes sample data including pixel value information of an image for learning and shape information of the region of interest.

[0009] A second aspect of the present invention provides an image processing method for estimating shape information, which is information regarding the shape of a region of interest, from an image, comprising the steps of acquiring a first image to be processed, acquiring information on two or more subspaces generated using different datasets, selecting information on a first subspace from the information on the two or more subspaces based on pixel value information of the first image, and estimating shape information of the region of interest in the first image from the pixel value information of the first image using the information on the first subspace, wherein the dataset includes sample data including pixel value information of an image for learning and shape information of the region of interest.

[0010] A third aspect of the present invention provides a program for causing a computer to function as each of the means of the image processing device relating to the first aspect described above, or a program for causing a computer to execute each of the steps of the image processing method relating to the second aspect described above, or a computer-readable storage medium non-temporarily storing such a program. Effect of the Invention

[0011] According to the present invention, information regarding the shape of a region of interest can be extracted from a medical image with higher accuracy. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a diagram showing the functional configuration of the image processing apparatus. [Diagram 2] 10 is a flowchart showing an example of a processing procedure of the image processing device. [Diagram 3] FIG. 2 is a diagram showing an example of an ultrasound image of the heart. [Figure 4] FIG. 2 is a diagram showing an example of a cloud of points representing the contour of a region of interest in an ultrasound image of the heart. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. However, the components described in the embodiment are merely examples, and the technical scope of the present invention is determined by the claims, and is not limited by the following individual embodiments.

[0014] An image processing device according to an embodiment of the present invention provides a function of estimating information about the shape of a region of interest (ROI) from an input image. The input image to be processed is a medical image, that is, an image of a subject (such as a human body) photographed or generated for the purpose of medical diagnosis, examination, research, etc., and is typically an image acquired by an imaging system called a modality. For example, ultrasound images obtained by an ultrasound diagnostic device, X-ray CT images obtained by an X-ray CT device, MRI images obtained by an MRI device, etc. can be processed. The input image may be a two-dimensional image or a three-dimensional image, Moreover, the image may be one of a time phase or a plurality of time phases. The region of interest is a part of an image, for example, an anatomical structure (organs, blood vessels, bones, etc.) or a lesion. What is selected as the region of interest can be set arbitrarily. Information on the shape of the region of interest (also referred to as "shape information" in this specification) is information that represents the spatial (or geometric) characteristics of the region of interest in the image. For example, the contour information of the region of interest (information that represents the contour shape), the positions of feature points on the region of interest and the relative positions (intervals, etc.) between feature points, the length, area, and volume of the region of interest, and the shape of the region of interest (circle, ellipse, triangle, etc.) can be exemplified. In addition to primary information obtained directly from an image, secondary information generated using primary information (for example, the difference in the positions of feature points, the area ratio between two time phases, etc.) is also included in the shape information.

[0015] The image processing device according to the embodiment of the present invention has one feature in that it obtains information on two or more subspaces generated using different data sets, and selects information on a subspace suitable for an input image from the information on the two or more subspaces to estimate shape information. When the learning data is divided into a plurality of data sets, the expressive power and generalization ability of the statistical model (subspace) may be inferior to the case where statistical analysis is performed on the entire learning data, but the risk of outputting an inappropriate estimation result is reduced. As the learning data, data in which each sample data includes pixel value information of an image for learning and shape information (correct answer data) of a region of interest in the image for learning can be used. By using such learning data, a model that well reflects the correlation between the pixel value information of an image and the shape information of a region of interest can be obtained, and the estimation accuracy can be improved.

[0016] A specific example of an image processing device according to an embodiment of the present invention will be described in detail below, taking as an example a case where the region of the left atrium of the heart is extracted from a two-dimensional ultrasound image.

[0017] The image processing device according to this embodiment has a function of automatically extracting contour information of the left atrium, which is a region of interest, from an input image. The image processing device according to this embodiment selects an appropriate subspace from two or more subspaces (statistical models) constructed from learning data according to the input image to be processed, and estimates shape information of the region of interest of the input image using the selected subspace.

[0018] The configuration and processing of the image processing device of this embodiment will be described below with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of an image processing system (also called a medical image processing system) including the image processing device of this embodiment. The image processing system includes an image processing device 10 and a database 22. The image processing device 10 is communicatively connected to the database 22 via a network 21. The network 21 includes, for example, a LAN (Local Area Network) or a WAN (Wide Area Network).

[0019] The database 22 holds and manages a plurality of images. The images managed by the database 22 include images whose contour information of the attention area is unknown and images whose contour information of the attention area is known. The former images are provided for contour extraction processing by the image processing device 10. The latter images are associated with contour information (ground truth data) and are used as learning data. It is desirable that these images are spatially normalized to a certain degree with respect to the size of the image and the attention area shown in the image. Spatial normalization is an operation of aligning the size (dimension per pixel) and position in the image to a certain standard. The information managed by the database 22 may include information on a subspace used in the contour extraction processing (for example, information on an estimated matrix). The information on the subspace may be stored in an internal memory (ROM 32 or memory unit 34) of the image processing device 10 instead of the database 22. The image processing device 10 can acquire data held in the database 22 via the network 21.

[0020] The image processing device 10 includes a communication IF (Interface) 31 (communication unit), a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33, and a The device includes a power supply unit 33, a memory unit 34, an operation unit 35, a display unit 36, and a control unit 37.

[0021] The communication IF 31 (communication unit) is composed of a LAN card or the like, and realizes communication between an external device (e.g., the database 22, etc.) and the image processing device 10. The ROM 32 is composed of a non-volatile memory or the like, and stores various programs and various data. The RAM 33 is composed of a volatile memory or the like, and is used as a work memory for temporarily storing programs and data being executed. The storage unit 34 is composed of a HDD (Hard Disk Drive) or the like, and stores various programs and various data. The operation unit 35 is composed of a keyboard, mouse, touch panel, etc., and inputs instructions from a user (e.g., a doctor or a medical technician) to various devices.

[0022] The display unit 36 ​​is configured with a display and the like, and displays various information to the user. The control unit 37 is configured with a CPU (Central Processing Unit) and the like, and controls the processing in the image processing device 10. The control unit 37 has, as its functional configuration, an image acquisition unit 51, a subspace information acquisition unit 52, a contour information estimation unit 53, and a display processing unit 54. The control unit 37 may also have a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), or the like.

[0023] The image acquisition unit 51 acquires a first image (an image with unknown contour information) to be processed from the database 22. That is, the image acquisition unit 51 corresponds to an example of an image acquisition means for acquiring the first image. The first image is an image of a subject acquired by various modalities. The image acquisition unit 51 may acquire the first image directly from the modality. In this case, the image processing device 10 may be implemented in the console of the modality (imaging system). In this embodiment, an example in which the first image is a two-dimensional ultrasound image will be described, but other types of images may also be used. The method of this embodiment is applicable to images with two or more dimensions (such as multiple two-dimensional images, two-dimensional moving images, three-dimensional still images, multiple three-dimensional images, or three-dimensional moving images). In addition, the method is applicable regardless of the type of modality.

[0024] The subspace information acquisition unit 52 acquires the learning data from the database 22, and performs two or more statistical analyses using the learning data. Each of the sample data constituting the learning data is data configured to include pixel value information of a learning image and shape information of a region of interest in the learning image (described in detail later). Then, the subspace information acquisition unit 52 calculates subspace information (for example, information on the bases constituting the subspace) representing the distribution of the sample data group from the result of the statistical analysis. At this time, the subspace information acquisition unit 52 divides the learning data into two or more data sets, and generates information on the two or more subspaces using different data sets. In other words, the subspace information acquisition unit 52 corresponds to an example of a subspace information acquisition means that acquires information on two or more subspaces generated using different data sets of the learning data.

[0025] The subspace information selection unit 55 selects first subspace information suitable for estimating contour information of the region of interest from the first image, based on the first image acquired by the image acquisition unit 51 and the two or more pieces of subspace information acquired by the subspace information acquisition unit 52. In other words, the subspace information selection unit 55 corresponds to an example of a subspace information selection means that selects the first subspace information from the information of two or more subspaces, based on the pixel value information of the first image.

[0026] The contour information estimation unit 53 estimates contour information of the left atrium in the first image acquired by the image acquisition unit 51 through a calculation using the first subspace information selected by the subspace information selection unit 55. That is, the contour information estimation unit 53 is an example of an estimation means that estimates shape information of the region of interest in the first image from pixel value information of the first image using the first subspace information. be.

[0027] Based on the results calculated by the contour information estimation unit 53, the display processing unit 54 displays the first image and the contour information of the estimated region of interest in the image display area of ​​the display unit 36 ​​in a display form that allows easy visual confirmation of the contour information.

[0028] Each of the components of the image processing device 10 functions according to a computer program. For example, the control unit 37 (CPU) reads and executes a computer program stored in the ROM 32 or the storage unit 34 using the RAM 33 as a work area, thereby realizing the function of each component. Note that some or all of the functions of the components of the image processing device 10 may be realized by using a dedicated circuit. Also, some of the functions of the components of the control unit 37 may be realized by using a cloud computer. For example, a calculation device located at a location different from the image processing device 10 may be communicably connected to the image processing device 10 via the network 21, and the image processing device 10 and the calculation device may transmit and receive data, thereby realizing the function of the components of the image processing device 10 or the control unit 37.

[0029] Next, an example of processing of the image processing device 10 in FIG. 1 will be described with reference to FIG. 2. FIG. 2 is a flowchart showing an example of a processing procedure of the image processing device 10. In this embodiment, an example in which the left atrium region of the heart is set as the region of interest will be described. However, this embodiment is also applicable to cases in which other parts including the left ventricle, right ventricle, and right atrium, or a region combining a plurality of these regions, is set as the region of interest.

[0030] (Step S101: Acquire and display image) In step S101, when the user issues an instruction to acquire an image via the operation unit 35, the image acquisition unit 51 acquires the first image designated by the user from the database 22 and stores it in the RAM 33. At this time, the display processing unit 54 may display the first image in the image display area of ​​the display unit 36. An example of the first image is shown in FIG. 3. FIG. 3 shows an example in which the first image is an apical four-chamber view in an ultrasound image of the heart. In the following description, it is assumed that the number of pixels in the x direction constituting the first image is Nx, and the number of pixels in the y direction is Ny. In other words, the total number of pixels constituting the first image is Nx×Ny.

[0031] (Step S102: Acquiring two or more pieces of subspace information) In step S102, the subspace information acquisition unit 52 acquires pixel value information (e.g., image data) of a plurality of images different from the first image and correct contour information of a region of interest in the image from the database 22. That is, the subspace information acquisition unit 52 acquires learning data used to calculate information on the subspace. Note that it is preferable that the plurality of images used as learning data are images of different patients in order to increase the robustness of the statistical model. However, images of the same patient taken at different times may be included. Alternatively, data augmentation (increasing the number of learning data) may be performed by deforming or processing the images.

[0032] Next, the subspace information acquisition unit 52 generates data that combines pixel value information and contour information of the region of interest for each of the multiple images acquired as learning data. Here, the pixel value information of the image and the contour information of the region of interest will be described in detail with reference to Figs. 3 and 4.

[0033] As shown in Fig. 3, the image handled in this embodiment is an image composed of Nx x Ny pixels. In this embodiment, the pixel value information of an image refers to a column vector in which the pixel values ​​of each pixel are arranged in the raster scan order of the image. That is, the upper left corner of the image shown in Fig. 3 is set as the origin (0,0), and the pixel value at the location of pixel (x, y) is represented as I(x, y), and the pixel value information a of the image is defined as follows:

number

[0034] Next, FIG. 4 shows an example in which the contour information of a region of interest is expressed by a group of nine points (p1 to p9) when the region is the left atrium region. In this embodiment, each of these points is also expressed by (x, y) using the coordinate values ​​of the x-axis and y-axis, with the upper left corner of the image being the origin (0, 0). Taking point p1 as an example, the coordinates of point p1 are (x1, y1). The contour information of the region of interest in this embodiment refers to a column vector in which the x-coordinate values ​​and y-coordinate values ​​of multiple feature points arranged on the contour of the region of interest are arranged. That is, in the example shown in FIG. 4, the contour information b representing the left atrium is defined as follows. Note that even when the "shape information" is the position of a feature point on the region of interest, b may be defined in a format in which the coordinate values ​​of the feature point are arranged in the same manner as above, such as b={x1, y1}. Furthermore, when the "shape information" is the length, area, or volume of the region of interest, b can be defined based on these values, such as b=length, b=area, and b=volume.

number

[0035] After acquiring the pixel value information a and the contour information b, the subspace information acquisition unit 52 further connects these column vectors to generate one column vector that combines the two pieces of information. That is, the subspace information acquisition unit 52 generates a column vector that includes an element corresponding to the pixel value information a and an element corresponding to the contour information b (shape information) of the region of interest. In the example shown in Figures 3 and 4, the following data c is generated.

number

[0036] Here, since the pixel value information of the image and the contour information of the region of interest have different variances, at least one of the data may be weighted. In this case, weighting may be performed so that the total variances are equal or balanced according to the total variances of the pixel value information and the contour information of the learning data. Alternatively, data may be weighted using weights set by the user via the operation unit 35.

[0037] The subspace information acquisition unit 52 generates data c by connecting pixel value information and contour information for each of the acquired multiple images (multiple learning sample data) in the above-mentioned manner. Then, the subspace information acquisition unit 52 divides the generated data group into two or more data sets, and performs statistical analysis on each data set. For the statistical analysis, a known method such as Principal Component Analysis (PCA) may be used, and methods such as Kernel PCA and Weighted PCA may also be used. By performing the principal component analysis, it is possible to calculate the average vector and eigenvectors for data c in which pixel value information and contour information are integrated, and the eigenvalues ​​corresponding to each eigenvector. That is, by using the average vector (c bar) of data c in which pixel value information and contour information are connected, the eigenvector e, and the coefficient g for each eigenvector e, it is possible to express a point d on the subspace by the following formula. This point d represents one learning sample data. The coefficient g is the principal component score.

number

[0038] Here, L represents the number of eigenvectors used in the calculation, and its specific value may be determined based on the cumulative contribution rate that can be calculated from the eigenvalue. For example, a cumulative contribution rate of 95% may be preset as a threshold, and the number of eigenvectors with a cumulative contribution rate of 95% or more may be calculated and set as L. Alternatively, the number of eigenvectors set by the user may be set as L.

[0039] As the final process of step S102, the subspace information acquisition unit 52 acquires subspace information corresponding to each data set from the result of the statistical analysis of each data set, and stores the information in the RAM 33. The subspace information is information that defines a subspace, and includes, for example, information on a plurality of eigenvectors and an average vector that configure the subspace. In this manner, in this embodiment, two or more pieces of subspace information having different bases are acquired based on the learning data.

[0040] It should be noted that the calculation process of the subspace information in this step is data processing independent of the contour extraction (contour estimation) of the first image. Therefore, the calculation process of the subspace information may be performed in advance, and the calculated subspace information may be stored in a storage device (for example, the database 22 or the storage unit 34). In this case, in step S102, the subspace information acquisition unit 52 may read two or more pieces of subspace information from the storage device and store them in the RAM 33. Calculating the subspace information in advance has the effect of shortening the processing time when performing the estimation process of the region of interest in the first image. It should be noted that the calculation process of the subspace information may be performed by a device other than the image processing device 10.

[0041] In addition, in the present embodiment, an example has been shown in which data in which pixel values ​​of the entire image are arranged in raster scan order is used as pixel value information, but other features related to pixel values ​​(such as features related to the texture of the image) may be used as pixel value information. Furthermore, data in which pixel values ​​of a partial image representing a part of an image are arranged may be used as pixel value information, or data in which only pixel values ​​around the contour of a region of interest in an image are arranged may be used as pixel value information.

[0042] Alternatively, data in which the principal component scores obtained by projecting an image onto a subspace obtained by performing principal component analysis on a plurality of images for learning are arranged may be used as pixel value information of the image. For example, after calculating pixel value information a for each of a plurality of images for learning, a principal component analysis may be performed on the data group of pixel value information a, and a vector of the principal component scores of each image may be used as new pixel value information a' of the image instead of a. When the total number of a plurality of images used as learning data is smaller than the number of pixels constituting the image, the number of dimensions of the pixel value information a' is smaller than the number of dimensions of the pixel value information a. Therefore, using the pixel value information a' has the effect of reducing the calculation cost of the statistical analysis of data in which pixel value information and contour information are linked. In addition, the calculation cost may be further reduced by setting a threshold value for the cumulative contribution rate to reduce the number of dimensions of the principal component scores (i.e., the number of eigenvectors).

[0043] On the other hand, as the contour information of the region of interest, the coordinate values ​​of the point group representing the contour of the region of interest are used, but other values ​​may be used. In one example, the results of calculating a level set function representing the region of interest (for example, a signed distance value from the contour with the inside of the region of interest being negative and the outside being positive) for each pixel of the image and arranging them in raster scan order may be used as the contour information. Alternatively, a label image or a mask image that distinguishes the region of interest from the rest may be used as the contour information. In addition, similar to the method of calculating the pixel value information a' from the pixel value information a described above, a value calculated using principal component analysis for the contour information b may be used as new contour information b'. In other words, by performing principal component analysis on a data group of only the contour information b corresponding to multiple images for learning and setting the vector of the principal component score of each image as the new contour information b', it is possible to further reduce the calculation cost.

[0044] Here, a specific example of a method for grouping the learning data into two or more data sets will be described. For example, a preset image transformation may be applied to an image acquired from the database 22 and contour information of a region of interest in the image, and grouping may be performed for each of the image transformations. That is, data augmentation (increasing the number of learning data) may be performed on image data related to the plurality of images and a point cloud representing the contour of the region of interest to create two or more data sets. More specifically, data may be generated by performing a plurality of operations such as translation, rotation, and enlargement / reduction, and data having the same or similar transformation parameters may be grouped together to create two or more data sets. Alternatively, grouping may be performed based on patient information (e.g., height, weight, or medical history) of the subject appearing in the image. Alternatively, grouping of the learning data may be performed based on differences in time phase (diastole and systole), differences in cross-sectional direction (four-chamber image and two-chamber image), differences in examination type or imaging protocol, differences in modality model, differences in hospital, differences in photographer, and the like. Grouping may also be performed by combining a plurality of these conditions. In addition, grouping may be performed using any classification that affects the shape or image quality of the region of interest.

[0045] Here, the effect of dividing the learning data into two or more groups, generating two or more pieces of subspace information, and selecting one subspace from them will be described. As shown in formula (4), in the results obtained from the principal component analysis, the subspace is expressed as a linear sum of the mean vector and the eigenvector. Here, when considering performing principal component analysis on the pixel values ​​of each pixel of the image, it is considered that in the above-mentioned subspace, an image that is plausible as image data (an image close to the learning data) is obtained in the peripheral part of the learning data of the analysis. However, for coordinates in the subspace that are far from the learning data, when the subspace is returned to the original image space by a backprojection operation, an unnatural image may be obtained. This is a problem that occurs because a linear model is used to model the relationship between pixel values ​​of different images that is not linear. If such a point exists in the subspace, the output may be inappropriate when an estimation process is performed using the subspace.

[0046] In general, it is desirable that the learning data to be subjected to statistical analysis has as many variations as possible, such as the shape of the region of interest and where the region of interest is located in the image. This is because the more variations in this data, the higher the expressive power of the subspace obtained from the statistical analysis, and the more unknown data can be handled. However, there is a problem that the higher the expressive power of the subspace, the higher the risk of the above-mentioned unnatural image being included in the subspace. In view of the above problem, in this embodiment, instead of generating one subspace using all sample data of the learning data, a method is adopted in which the learning data is divided into multiple data sets that collect relatively close sample data, and then a subspace is generated for each of these data sets. That is, according to this method, a subspace specialized for a group of learning sample data having a certain characteristic can be generated according to the data classification method. By dividing the learning data into multiple data sets, the expressive power of the subspace for one set is reduced compared to the case of generating one subspace as a whole, but the effect of reducing the risk of an unnatural image being included in the subspace can be expected. In the method of this embodiment, when an unknown input image is given, it is important to select which subspace information to use for estimation. Therefore, in the next step S103, subspace information suitable for the estimation process is selected.

[0047] (Step S103: Selection of subspace information) In step S103, the subspace information selection unit 55 selects one piece of subspace information based on the first image acquired by the image acquisition unit 51 and the two or more pieces of subspace information acquired by the subspace information acquisition unit 52, and stores the result in the RAM 33. That is, the subspace information (the first subspace information) more suitable for estimating the contour information of the region of interest in the first image is selected. As the subspace information, one piece of subspace information is selected from two or more pieces of subspace information based on the first image.

[0048] The process of selecting subspace information suitable for estimating contour information of the region of interest from two or more pieces of subspace information can be realized by using the similarity between the pixel value information of the first image and the pixel value information of the sample data included in each data set. That is, the subspace information selection unit 55 may specify a data set including sample data similar to the first image based on the pixel value information of the image, and select the subspace information generated from the specified data set.

[0049] Here, a specific method for identifying one dataset that is optimal for the first image from two or more datasets includes a method of evaluating the difference between the first image and each sample image. The difference between two images may be evaluated, for example, by the sum of squared differences (SSD) of pixel values. For example, the sample data that has the smallest sum of squared differences is considered to be the sample data that is most similar to the first image (referred to as similar sample data), and the dataset that includes the similar sample data is identified as the dataset that is optimal for the estimation process. By using the subspace information generated from the dataset that includes the sample data that is most similar to the first image in this way, it is expected that the accuracy of the estimation process in the subsequent step S104 will be improved. However, this method has a problem in that when the total number of sample data is huge, the calculation cost becomes huge.

[0050] Therefore, in this embodiment, the subspace information selection unit 55 performs a principal component analysis on the pixel value information of the sample data of each data set, and then identifies the optimal data set by the subspace method using the result of the principal component analysis and the pixel value information of the first image. This method can reduce the calculation cost of the above-mentioned identification process.

[0051] The procedure for processing by the subspace method is as follows. First, a principal component analysis is performed on pixel value information for each of two or more data sets to generate a subspace. Then, for each of all subspaces, a process of projection and backprojection of the first image into the subspace is performed to calculate a reconstruction error. That is, after projecting the pixel value information of the first image into the subspace, the sum of squared pixel value differences (i.e., reconstruction error) is calculated between the image obtained from the result of backprojection into the original image space and the first image. The data set corresponding to the subspace with the smallest reconstruction error is the data set including sample data similar to the first image, in other words, the data set that provides the subspace that can most accurately express the first image. By this method, a data set corresponding to the subspace information most suitable for the process in step S104 in the subsequent stage can be identified.

[0052] Although the example shown here uses the sum of squared differences of pixel values ​​to evaluate the similarity between images, other indices such as the sum of absolute differences (SAD) may also be used.

[0053] Moreover, the present invention is not limited to selecting one optimal dataset from among a plurality of datasets, and may be configured to select all datasets that satisfy a predetermined condition. For example, all datasets in which the reconstruction error of pixel value information is smaller than a predetermined threshold value may be selected. Alternatively, a predetermined number of datasets in which the reconstruction error is smallest may be selected. When a plurality of datasets are selected, in step S104, the contour information estimation unit 53 may execute a contour information estimation process using the subspace information of each dataset and integrate the results. For example, the average value or median value of the contour information estimated using each subspace information may be acquired.

[0054] (Step S104: Estimation of contour information of the region of interest) In step S104, the contour information estimation unit 53 estimates contour information of the attention area in the first image from the first image acquired in step S101 and the first subspace information selected in step S103. The estimation result is stored in the RAM 33.

[0055] Any method for estimating the contour information may be used. For example, Non-Patent Document 1 discloses a technique for estimating contour information of a region of interest in an input image based on iterative processing using a gradient method, using an evaluation value that indicates the consistency between pixel value information of an unknown input image and pixel value information of an image in subspace information (statistical model). The contour information estimation unit 53 may estimate the contour information using such iterative processing using a gradient method.

[0056] However, when using iterative processing to estimate contour information, it is necessary to repeat the iterative processing for a predefined energy function until the change before and after the iterative processing becomes sufficiently small or until a certain number of iterations is reached, which incurs a certain amount of calculation cost. In addition, since the number of iterations (the timing at which the processing converges) varies depending on the input data, there is also the issue that it is difficult to grasp in advance the calculation time required for the processing due to variations in the overall calculation time.

[0057] Therefore, in this embodiment, a method that does not use iterative processing is applied when estimating contour information of a region of interest, thereby achieving estimation processing of contour information with lower calculation costs.

[0058] Here, the problem of estimating contour information representing a predetermined region of interest (left atrium) when an unknown input image (ultrasound image) is given will be summarized. The first subspace information selected in the previous step S103 is information on a subspace that associates pixel value information of the ultrasound image with information on a point cloud representing the contour of the left atrium shown in the image. When dealing with an unknown input image, an ultrasound image is given, so that pixel value information a corresponding to the image can be calculated in the same manner as in step S102. In other words, the proposition in this step is to estimate contour information b corresponding to the input image based on the first subspace information calculated from the learning data and pixel value information a related to the input image.

[0059] Non-Patent Document 2 below discloses a technique for estimating information representing the posture of an object appearing in an image from pixel value information of an unknown image by using subspace information related to data that combines pixel value information of a plurality of images and information representing the posture of the object appearing in the image. In other words, this technique is a technique for interpolating data of a missing portion from the result of statistical analysis of learning data without a missing portion for data with a missing portion. This technique is called the Back Projection for Lost Pixels (BPLP) method.

[0060] [Non-Patent Document 2] Toshiyuki Amano,et.al.”An appearance based fast linear pose estimation” MVA 2009 IAPR Conference on Machine Vision Applications.2009 May 20-22.

[0061] In order to apply the BPLP method, it is necessary to specify which parts of the input data are known information and which parts are unknown information (missing parts) at the time of estimation. In this embodiment, the pixel value information of the first image, which is the input image, is set as known information, and the contour information of the region of interest in the first image is set as unknown information. That is, the problem setting is changed so that the part of the information on the pixel values ​​of the image in Non-Patent Document 2 is pixel value information a, and the part of the information representing the posture of the object is contour information b, and the proposition of this step is solved based on the following method.

[0062] The contour information estimation unit 53 obtains a vector f including an element corresponding to pixel value information a of the first image, which is known information, and an element corresponding to contour information b of the region of interest in the first image, which is unknown information, by the following formula. Since the calculation based on the following formula does not include repeated calculations, it is possible to realize a contour information estimation process with a lower calculation cost than the conventional gradient method. Furthermore, since the factors that affect the variation in the calculation time for this calculation are the input pixel value information and the number of dimensions of the contour information to be estimated, it is expected that the variation in the processing time will be small once the size of the input image is determined.

number

[0063] Here, f' on the right-hand side represents input data containing a certain loss, and in this embodiment, as shown in equation (3), the elements of vector f that correspond to unknown contour information are set to 0.

number

[0064] E is a matrix representing a subspace defined by the first subspace information selected in step S103. When L eigenvectors are e1, e2, . . . , eL, for example, the matrix E is given by E = [e1, e2, . . . , eL]. Σ is a square matrix in which diagonal elements corresponding to pixel value information, which is known information, are set to 1, and other elements are set to 0. In other words, Σ is a matrix in which elements of a unit matrix corresponding to contour information, which is unknown information, are set to 0. In this embodiment, Σ is a square matrix with a size of one side of Nx × Ny + 9 × 2 (the number of dimensions of pixel value information a + the number of dimensions of contour information b), and the last 18 diagonal elements are 0, and the other diagonal elements are 1.

number

[0065] In this embodiment, even if the input image is arbitrary, the part set as unknown information does not change (it is always the part of the contour information b). Therefore, at the time when the subspace information is calculated in step S102, E (E T ΣE) -1 E T (Part of the calculations performed in the estimation process) can be calculated in advance. T ΣE) -1 E T The results of the calculations may be stored as subspace information in a storage device (for example, the database 22 or the storage unit 34). T ΣE) -1 E T If the result of E(E T ΣE) -1 E T Some of them (for example, E T ΣE) -1The calculation results of the above part (part A) may be stored in a storage device (for example, the database 22 or the storage unit 34) as subspace information, and the calculation of the remaining part may be performed by the subspace information acquisition unit 52 or the contour information estimation unit 53.

[0066] Finally, the contour information estimation unit 53 extracts information of the portion corresponding to the contour information from the estimation result f (in this embodiment, the last 18 elements), and stores it in the RAM 33.

[0067] In addition, when pixel value information a' is used instead of pixel value information a in step S102, it is necessary to replace I(0,0) to I(Nx,Ny) in the formula (6) with values ​​calculated in the same manner as the pixel value information a'. More specifically, when pixel value information a' is used for learning, In the case of a principal component score related to pixel value information of an image, a projection process is performed based on the input image and information on a subspace constructed using only pixel value information of the learning data to calculate a principal component score of the input image. In addition, in the case of performing an estimation process on values ​​such as contour information b' as a target of the estimation process, the contour information of the estimation result is not in xy coordinates on the contour line, so that the estimated values ​​are converted into coordinate values ​​and stored in the RAM 33. More specifically, in the case of contour information b' being a principal component score related to the contour information of the learning data, a back projection process is performed based on the principal component score and information on a subspace constructed using only contour information of the learning data.

[0068] The contour information of the estimated region of interest may be used as it is as a contour estimation result, or may be used as a rough extraction result to be given as an initial value for a more precise extraction process. In the latter case, the contour information estimation unit 53 uses the contour information obtained by the above process as an initial value and performs precise contour extraction using a known method for precisely extracting an object contour.

[0069] (Step S105: Display image based on contour information estimation result) In step S105, the display processing unit 54 displays the first image, which is the input image, and the contour information of the region of interest estimated by the contour information estimation unit 53 in the image display area. At this time, the estimated contour information and the first image may be displayed in a superimposed manner. By performing the superimposed display, it is possible to easily visually confirm to what extent the estimated contour information matches the first image. In this embodiment, the contour information of the region of interest is a discrete point group obtained by sampling the contour of the region, and therefore, it may be displayed after interpolating between adjacent points using a known technique such as spline interpolation.

[0070] When the purpose is to analyze or measure a region of interest, the process of step S105 is not necessarily required, and the configuration may be such that the estimated contour information is simply stored.

[0071] Furthermore, in the above description, the image is a two-dimensional image, but the same processing can be performed even if the image is a three-dimensional image (volume image).

[0072] In this embodiment, the coordinate values ​​of the point cloud representing the left atrium are used as the contour information of the region of interest, but the contour information may be a combination of coordinate values ​​of point clouds representing two or more regions, such as the left ventricle, right ventricle, and right atrium in addition to the left atrium. In this case, by performing statistical analysis on not only the coordinate values ​​of the point cloud representing the left atrium region but also the coordinate values ​​of all the point clouds of the two or more regions in step S102, the coordinate values ​​of all the point clouds of the two or more regions can be simultaneously estimated in step S104.

[0073] According to this embodiment, by selecting an optimal subspace from among a plurality of subspaces according to the first image and estimating contour information of the region of interest, it is possible to provide a user with a more accurate contour extraction result. Also, according to this embodiment, by estimating contour information, which is shape information of the region of interest, by a matrix operation using a matrix E representing a subspace constructed from learning data, it is possible to provide a user with a more accurate contour extraction result at a lower calculation cost.

[0074] Although the present embodiment has been described above, the present invention is not limited to the above, and can be modified and changed within the scope of the claims.

[0075] (Variation 1) In the above embodiment, an example was shown in which the images treated as input images or learning data were two-dimensional ultrasound images, but these images may be images (three-dimensional time-space images) in which data of two-dimensional images are linked over time. In other words, a two-dimensional ultrasound image video (two or more frames of two-dimensional ultrasound images) may be handled as an input image. In this case, a plurality of frames consecutive in time may be used. A three-dimensional spatio-temporal image may be constructed using a set of images of a specific frame (phase) rather than a set of images of all frames.

[0076] Taking a moving image of an ultrasound image of the heart as an example, a three-dimensional image composed of two time-phase images, an image of a frame of diastole (ED (End-Diastole) phase) and an image of a frame of systole (ES (End-Systole) phase). In this case, the method of the above embodiment may be performed based on pixel value information of the two frame images of diastole and systole and contour information corresponding to each of the images. By performing the same process as the above embodiment, contour information corresponding to all frames of an unknown three-dimensional image can be simultaneously estimated for the input of the three-dimensional image. Note that any known method may be used to identify the frames of the diastole and systole of the heart. For example, the frames may be manually identified by a user from the moving image, or the frames may be automatically identified from information such as an electrocardiogram corresponding to the moving image.

[0077] When constructing a three-dimensional image from multiple frames as described above, in order to implement the method of the first embodiment, all pixel values ​​of all frames of input images are set as pixel value information. That is, for example, when a three-dimensional image is constructed from diastolic (di) and systolic (sys) images, pixel value information a is set as follows:

number

[0078] Here, I di (x,y) represents the pixel value of the diastolic image, and I sys (x, y) represents the pixel value of the image during systole. In other words, when the input image is an image composed of a set of images of multiple frames (time phases), data obtained by concatenating pixel value information of the images of all frames can be used as pixel value information of the input image.

[0079] Furthermore, for contour information, the method of the above embodiment can be implemented by using data obtained by concatenating information on a group of points representing contour information of a region of interest corresponding to each image of all frames as contour information of a region of interest corresponding to an input image. That is, the contour information b can be as follows. Here again, the subscript di indicates the coordinates of a point representing the contour information of a region of interest in an image during diastole, and the subscript sys indicates the coordinates of a point representing the contour information of a region of interest in an image during systole.

number

[0080] At this time, in step S101, the image acquisition unit 51 acquires an image pair of diastole and systole to be processed as a first image. In addition, in step S102, the subspace information acquisition unit 52 performs the process described in the first embodiment for a plurality of image pairs of diastole and systole different from the first image. That is, a statistical analysis is performed on data c obtained by linking pixel value information a (Equation (8)) of the image pair and correct contour information b (Equation (9)) of the region of interest in the image pair, and subspace information is acquired. Then, in step S103, the contour information estimation unit 53 estimates contour information of the region of interest appearing in the first image. In this way, by setting the pixel value information and the contour information and then performing the same process as in the above embodiment, it is possible to simultaneously estimate contour information corresponding to all frames of an unknown three-dimensional image when the three-dimensional image is input.

[0081] As described in the above embodiment, the principal component analysis may be performed on the data of only the pixel value information of the image, the principal component score corresponding to each image may be calculated, and the principal component score may be used as new pixel value information for each image. If the pixel value information is composed of values ​​on a pixel-by-pixel basis as in Equation (8), As the number of frames of a 3D image increases, the number of dimensions of the data becomes enormous, leading to an increase in calculation costs. Therefore, by using principal component scores with a small number of dimensions as pixel value information, calculation costs can be significantly reduced. Similarly, for contour information, principal component scores calculated using principal component analysis of the coordinate values ​​of points representing the region of interest can be used as new contour information.

[0082] In this way, extracting two or more frame images, such as diastole and systole, from a video sequence to construct a 3D image and then estimating the contour information of these two or more frame images simultaneously has the following two advantages: These effects can be obtained not only when combining diastole and systole images, but also by using a combination of multiple images of different time phases.

[0083] The first advantage is the effect of stabilizing the estimation accuracy of the region of interest. In other words, the robustness of the estimation process is improved by using multiple frames. In the moving images of ultrasound diagnostic images, the image quality (degree of contrast and noise) of each frame often varies. In other words, even if the contour of the region of interest is clearly depicted and highly visible in one frame of the moving image, the visibility of the contour may be low in another frame. Therefore, when extracting a certain frame from the moving image and performing the region estimation process, a frame with low image quality may become the processing target, and there is a risk that the extraction accuracy will decrease if the region extraction process is performed on the image with low image quality. In such a case, in order to improve the region extraction accuracy in the image with low image quality, a method of using an image with high image quality that is correlated with the image can be considered. As a premise, it is necessary that the input three-dimensional image contains a frame with high image quality, but by performing the region extraction process based on multiple frames, it is expected that the average extraction accuracy will be improved and the variation in the extraction accuracy will be reduced. In other words, it is expected that the robustness of the estimation process will be improved.

[0084] The second advantage is that it is possible to perform region estimation processing that takes into account the relationship between the position and size of the region of interest between different frames. Taking an image capturing the movement of the heart as an example, it is conceivable that there is a correlation between the size of the atrium and ventricle and the location where their anatomical structures are captured in the image between the diastole and the systole. However, if the estimation processing of the region of interest is performed independently for different frames, the processing is performed without taking into account the relationship between the frames, and therefore the relationship between the frames, such as the size relationship of the region of the estimation result, may be lost. For example, even if the area of ​​the region of interest in the diastole should be larger than that in the systole, the estimated result in the diastole may be smaller. By expanding the above embodiment to handle images of multiple frames, a subspace that takes into account the relationship between the regions of interest between different frames is obtained, and therefore, an output result that maintains the original relationship between the frames is obtained for the estimation results between different frames.

[0085] In this modified example, the contour information of the region of interest is estimated by a matrix calculation using a matrix representing a subspace constructed from learning data, so that a highly accurate contour extraction result can be provided to the user at a lower calculation cost. The combinations constituting the three-dimensional image may be combinations of past and current images as well as combinations of different frames (phases) in a single examination. The combinations may also be combinations of two-dimensional images in multiple cross-sectional directions (for example, four-chamber and two-chamber images, or long-axis and short-axis images) as well as combinations of different times. The combinations may also be combinations of cross-sectional images in multiple frames and directions.

[0086] (Variation 2) In the above embodiment and modification 1, pixel value information of an input image is set as known information, and contour information of a region of interest corresponding to the input image is set as unknown information, and the unknown information is estimated by the BPLP method. However, the known information and the unknown information may be set differently. .

[0087] For example, in the above embodiment, when at least a part of the contour information of the region of interest is known in advance in addition to the pixel value information of the input image, the contour information may be set as known information, and the other contour information may be set as unknown information. For example, the position of the annulus in the region of interest may be detected using a method such as known feature point extraction, and the position may be set as known information. In addition, the user may manually set the position of the annulus on the image via the operation unit 35. For example, if point p1 in the contour information b is the position of the annulus, (x1, y1) can be given as known information. That is, in step S103, the contour information estimation unit 53 gives the coordinates of the acquired annulus position to the elements corresponding to x1, y1 in equation (6), changes the values ​​of the diagonal elements corresponding to x1, y1 in equation (7) to 1, and estimates the coordinates of p2 to p9 using equation (5).

[0088] In addition, in the first modification, an example was shown in which pixel value information of a three-dimensional image is set as known information, and contour information of a region of interest corresponding to each frame of the three-dimensional image is set as unknown information, and the unknown information is estimated by the BPLP method. Even in this case, when at least a part of the contour information of a region of interest in a certain frame is known, the contour information may be set as known information. For example, when a three-dimensional image composed of images in the diastole and the systole is input, and the contour information in the diastole is known in advance or the contour information is specified by the user, only the contour information in the systole may be set as unknown information. Increasing the known information in this way increases the information that can be used in the estimation process, thereby improving the accuracy of the estimation process of the unknown information.

[0089] In another example, pixel value information of an image may be estimated by the BPLP method based on contour information input by a user via the operation unit 35 or contour information generated using subspace information of the contour information of a region of interest, etc. In this case, if the contour information is set as known information and the pixel value information of the image is set as unknown information, the estimation process can be performed in the same procedure as in the above embodiment.

[0090] In this modified example, unknown information is estimated by a matrix operation using a matrix representing a subspace constructed from training data, so that the effect of being able to provide an estimation result to a user at low computational cost is maintained.

[0091] The above is one example of an embodiment, but the present invention is not limited to the embodiment described above and shown in the drawings, and can be modified appropriately without departing from the spirit and scope of the present invention.

[0092] <Other embodiments> The disclosed technology can be embodied as, for example, a system, an apparatus, a method, a program, or a recording medium (storage medium), etc. Specifically, the technology may be applied to a system composed of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to an apparatus composed of a single device.

[0093] Needless to say, the object of the present invention can be achieved by the following: A recording medium (or storage medium) on which is recorded software program code (computer program) for realizing the functions of the above-mentioned embodiments is supplied to a system or device. The storage medium is, of course, a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the recording medium. In this case, the program code itself read out from the recording medium will realise the functions of the above-mentioned embodiments, and the program code itself will be read out from the recording medium and executed. The recording medium on which the program code is recorded constitutes the present invention. [Explanation of symbols]

[0094] 10 Image processing device 51 Image acquisition unit 52 Subspace information acquisition unit 53 Contour information estimation unit 54 Display processing section 55 Subspace information selection unit

Claims

1. An image processing device that estimates shape information, which is information about a shape of a region of interest, from an image, comprising: an image acquisition means for acquiring a first image to be processed; A subspace information acquisition means for acquiring information on two or more subspaces generated using different data sets; a subspace information selection means for selecting information of a first subspace from information of the two or more subspaces based on pixel value information of the first image; an estimation means for estimating shape information of a region of interest in the first image from pixel value information of the first image by using information of the first subspace, The image processing device according to claim 1, wherein the data set includes sample data including pixel value information of an image for learning and shape information of a region of interest.

2. The image processing apparatus according to claim 1 , wherein the information on the subspace represents a distribution of sample data included in the data set.

3. 3. The image processing device according to claim 1, wherein the subspace information selection means identifies a dataset including sample data similar to the first image based on a similarity between pixel value information of the first image and pixel value information of sample data included in each dataset, and selects information of a subspace generated from the identified dataset as information of the first subspace.

4. 4. The image processing device according to claim 3, wherein the subspace information selection means evaluates the similarity between the first image and each data set based on a reconstruction error when pixel value information of the first image is projected and backprojected onto a subspace related to pixel value information of each data set.

5. 4. The image processing device according to claim 1, wherein the information on the subspace is information obtained by performing a statistical analysis on a plurality of sample data included in the data set.

6. The statistical analysis is a principal component analysis; 6. The image processing apparatus according to claim 5, wherein the information on the subspace includes information on a mean vector and a plurality of eigenvectors obtained by principal component analysis.

7. The image processing device according to any one of claims 1 to 6, characterized in that the subspace information acquisition means acquires information on the two or more subspaces from a storage device that stores information on subspaces generated in advance.

8. 8. The image processing apparatus according to claim 1, wherein the first image is an image constituted by a set of a plurality of images in different time phases.

9. The image processing device according to claim 8, characterized in that the estimation means uses data obtained by concatenating pixel value information of the multiple images at different time phases as pixel value information of the first image, and uses data obtained by concatenating shape information corresponding to each of the multiple images at different time phases as shape information corresponding to the first image.

10. the first image is an image of the heart; The region of interest is an atrium or a ventricle.

2. The image processing device according to claim 1 .

11. 11. The image processing apparatus according to claim 10, wherein the first image is a set of an image of the heart in a diastole and an image of the heart in a systole.

12. 12. The image processing apparatus according to claim 1, wherein the pixel value information is data in which pixel values ​​of an image are arranged.

13. The image processing device according to any one of claims 1 to 12, characterized in that the pixel value information is data that lists principal component scores of an image, obtained by projecting the image onto a subspace obtained by principal component analysis of multiple images for learning.

14. 14. The image processing apparatus according to claim 1, wherein the shape information is contour information of the region of interest.

15. An image processing method for estimating shape information, which is information about a shape of a region of interest, from an image, comprising: obtaining a first image to be processed; obtaining information on two or more subspaces generated using different data sets; selecting a first subspace information from the two or more subspace information based on pixel value information of the first image; and estimating shape information of a region of interest in the first image from pixel value information of the first image using information of the first subspace, The image processing method according to the present invention, characterized in that the data set includes sample data including pixel value information of a learning image and shape information of a region of interest.

16. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 14.

Images