Image processing apparatus, image processing method, and program

Abstract

To provide a technique capable of displaying a fracture part three-dimensionally and easily visibly even when a comparatively large range of space including the fracture part is estimated as a lesion space.SOLUTION: An image processing device displays a lesion bone part area 77 (a curve surface area 79 and the like) being a product space of an estimated lesion space 75 (including a neighboring space adjacent to a bone of a subject) estimated as a space including a fracture part 50 based on a group of two-dimensional slice images of the subject and a portion including at least the surface of a three-dimensional bone model 330 being a three-dimensional model of the bone of the subject, on a display unit in a mode different from a portion (332 and the like) other than the lesion bone part area 77 in the three-dimensional bone model 330.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and related technologies.

Background Art

[0002] There is a technology for acquiring a plurality of two-dimensional slice images (cross-sectional images) of a subject using an MRI (Magnetic Resonance Imaging) apparatus or a CT (Computed Tomography) apparatus, and identifying a lesion using the plurality of two-dimensional slice images.

[0003] For example, there is a technology for generating volume data based on a plurality of two-dimensional slice images and determining the presence or absence and location of a fracture (lesion) based on the volume data (see Patent Document 1). In Patent Document 1, it is described that a region where there is a left-right difference (a region where rib deformation exists) is detected as a rib fracture region by utilizing the left-right symmetry regarding the three-dimensional shape of the rib, and the rib fracture region is displayed (see paragraph 0074 and FIG. 13 of Patent Document 1). In Patent Document 1, when a fracture region of one of a pair of left and right ribs is detected, a VR (Volume Rendering) image (volume rendering image) from the front direction near the rib is generated, and the fracture region on the VR image is emphasized and displayed.

[0004] There is also a technology for detecting a lesion site from a plurality of two-dimensional slice images using a machine-learned learning model.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In Patent Document 1, it is shown that a region where a left - right difference (positional deviation) exists is detected as a rib fracture region by utilizing the left - right symmetry regarding the three - dimensional shape of the rib. However, for the detection of a fracture site (lesion site), not only the detection technique as in Patent Document 1 is used, but also a detection technique using machine learning as described above may be used.

[0007] In particular, as a detection technique for a fracture site using machine learning, there is also a technique of detecting a region including the fracture site (a region wider than the fracture site alone) as a presumed lesion region (a region presumed to be a lesion region), rather than a region composed only of the fracture site. For example, a region including the neighboring space adjacent to the bone of the subject may be detected as the presumed lesion region. According to the detection technique for a fracture site using machine learning, since the fracture site can be detected regardless of the left - right symmetry (left - right shape difference) regarding the three - dimensional shape, it is possible to appropriately detect various types of fractures. For example, it is possible to detect not only complete fractures (such as fractures where the bone is broken and separated into multiple parts) but also incomplete fractures (such as fractures with cracks in the bone).

[0008] And even when such a presumed lesion region is detected, it is required to three - dimensionally display the position of the lesion in three - dimensional volume data (for example, a bone model of the subject).

[0009] In response to such a requirement, as will be described later, for example, it is conceivable to directly superimpose and display a space (also referred to as a presumed lesion space) formed by stacking the presumed lesion regions in a plurality of two - dimensional slice images on the bone model of the subject. Specifically, it is conceivable to represent the presumed lesion space as a solid model and display it in a state where the solid model is combined with the bone model.

[0010] However, the estimated lesion space formed by stacking a relatively wide estimated lesion area (the area including the fracture site) often has a portion protruding outward from the surface of the bone model. Therefore, in a display where the estimated lesion space is directly superimposed on the bone model of the subject, the surface of the bone model may be covered by the estimated lesion space and become invisible, making it difficult to grasp the fracture position and the like in the bone model.

[0011] Therefore, an object of the present invention is to provide a technique capable of three-dimensionally and easily displaying a fracture site even when a relatively wide space including the fracture site is estimated as a lesion space.

Means for Solving the Problems

[0012] To solve the above problems, an image processing apparatus according to the present invention is based on a two-dimensional slice image group of a subject of the subject an estimated lesion space estimated as a space including a fracture site having the fracture site adjacent to the bone and protruding outward from the surface of the bone an estimated lesion space including a neighboring space, and a lesion bone region which is a product space of at least a portion including the surface of a three-dimensional bone model which is a three-dimensional model of the bone of the subject, among the three-dimensional bone model 、 a control unit that displays the lesion bone region on a display unit in a manner different from a portion other than the lesion bone region.

[0013] The control unit may estimate the estimated lesion space using a learned model learned by machine learning.

[0014] The control unit estimates, as an estimated lesion region, a two-dimensional region including a fracture site in each of two or more two-dimensional slice images of the two-dimensional slice image group of the subject using a learned model learned by machine learning, and may obtain the product space of the estimated lesion space and the surface of the three-dimensional bone model based on the estimated lesion regions of the two or more two-dimensional slice images.

[0015] The control unit may change the display mode of each position in the lesion bone region, which is the product space of the estimated lesion space and at least the part including the surface of the three-dimensional bone model, according to the reliability regarding the lesion estimation at each position.

[0016] The control unit may generate stage-by-stage data obtained by classifying the lesion bone region, which is the product space of the estimated lesion space and at least the part including the surface of the three-dimensional bone model, into a plurality of stages according to the reliability regarding the lesion estimation, and output the stage-by-stage data to different data files for each stage.

[0017] The control unit may change the display mode of the lesion bone region, which is the product space of the estimated lesion space and at least the part including the surface of the three-dimensional bone model, according to the type of the lesion.

[0018] In order to solve the above problems, the image processing method according to the present invention includes: a) based on a two-dimensional slice image group of a subject of the subject An estimated lesion space estimated as a space including a fracture site having the fracture site Adjacent to the bone and protruding outward from the surface of the bone Including a neighboring space, and at least a part including the surface of a three-dimensional bone model, which is a three-dimensional model of the bone of the subject, a lesion bone region, which is a product space of the two, is displayed on a display unit in a manner different from a part other than the lesion bone region among the three-dimensional bone models. 、 characterized by comprising the step of: To solve the above problems, the program according to the present invention causes a computer to execute a step of displaying, on a display unit, in a manner different from a portion other than the diseased bone region in the three-dimensional bone model, a diseased bone region which is a product space of an estimated lesion space estimated as a space including a fracture site of the subject and including a neighborhood space adjacent to a bone having the fracture site and protruding outward from the surface of the bone, and at least a portion including the surface of a three-dimensional bone model which is a three-dimensional model of the bone of the subject. The program is characterized by this.

Advantages of the Invention

[0019] According to the present invention, even when a wider space including a fracture site is estimated as a lesion space, it is possible to three-dimensionally and easily display the fracture site.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Mode for Carrying Out the Invention

[0021] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0022] <1. First Embodiment> <1-1. System Overview> FIG. 1 is a block diagram showing an image processing system 10. The image processing system 10 is a system that processes a plurality of two-dimensional slice images (tomographic images) 220 (see FIG. 4) (specifically, a plurality of two-dimensional slice image groups 210) obtained by slicing a subject (such as a subject) in a cross-section perpendicular to the reference axis.

[0023] As shown in FIG. 1, the image processing system 10 includes a slice image generation device 20 and an image processing device 30. The slice image generation device 20 and the image processing device 30 are connected by wire (or wirelessly) and can communicate with each other. The image processing device 30 receives information (such as slice images) generated (acquired) by the slice image generation device 20 from the slice image generation device 20 via a predetermined connection cable or the like.

[0024] The slice image generation device 20 is composed of an MRI (Magnetic Resonance Imaging) device or a CT (Computed Tomography) device, etc. The slice image generation device 20 generates and acquires a plurality of two-dimensional slice images (also simply referred to as slice images) 220 regarding a subject (such as a subject). The plurality of slice images 220 are images obtained by slicing the subject in a cross-section perpendicular to the reference axis at a plurality of different positions on the reference axis (for example, at different positions with a pitch of 0.6 mm to 1 mm (millimeter)). The plurality of slice images 220 are acquired over a predetermined range (for example, 300 mm) in the reference axis direction and are composed of several hundred to several thousand (for example, 500) images. In the slice image generation device 20, for example, a two-dimensional slice image group 210 (a plurality of two-dimensional slice images 220) regarding the lumbar region including the pelvis is imaged and acquired (see FIG. 4).

[0025] The image processing device 30 executes an inference process for each of a plurality of slice images 220 (also referred to as 240) obtained for a certain subject, using a learning model 410 (also referred to as a learned model 420) whose learning parameters have been adjusted in advance (see FIG. 2). FIG. 2 is a conceptual diagram showing the processing in the inference stage in machine learning.

[0026] Specifically, the image processing device 30 executes an inference process for estimating the (unknown) lesion site (fracture site) of each slice image 240 using the learned model 420 described above. Here, a fracture site (more specifically, a fracture site in the pelvis) is mainly exemplified as the lesion site. In this inference process, for example, the presence or absence of a fracture site and the position of the fracture site are specified (estimated). Specifically, a region including the fracture site 50 (see FIG. 6) (more specifically, a region including the vicinity region (vicinity region adjacent to the bone) of the fracture site 50) is specified as the estimated lesion region 71 (see FIG. 5). For example, a two-dimensional bounding box surrounding the fracture site 50 is specified as the estimated lesion region 71. Note that the estimated lesion region 71 is a region estimated to be a region including a lesion (fracture site).

[0027] In addition, the reliability (for example, 80%) regarding the inference result of the fracture site is also output. The reliability may be determined for each slice image 240, or may be determined for each pixel in the slice image 240.

[0028] Note that the learned model 420 is generated by adjusting the learning parameters of the learning model 410 (learner) using a predetermined machine learning method. As the learning model 410, for example, a neural network model composed of a plurality of layers is used. Then, by a predetermined machine learning method (such as deep learning), the weighting coefficients (learning parameters) between the plurality of layers (input layer, (one or more) intermediate layers, output layer) in the neural network model are adjusted. The processing of this learning stage in such machine learning is, for example, executed in advance by the image processing apparatus 30. Specifically, a learning model 410 (learned model 420) for executing an inference process for identifying a lesion site in a slice image is generated based on teacher data having a lesion site (a plurality of slice image groups having known lesion sites (more specifically, a plurality of slice image groups related to a plurality of specimens)). However, it is not limited thereto, and the image processing apparatus 30 may construct the learned model 420 in its own apparatus (image processing apparatus 30) by acquiring learning parameters adjusted by another apparatus.

[0029] In addition, the image processing apparatus 30 executes further processing based on the inference result using the learned model 420 by machine learning.

[0030] Specifically, as will be described later, the image processing apparatus 30 virtually forms an estimated lesion space 75 based on the estimated lesion region 71. The estimated lesion space 75 is a space estimated as a space including the fracture site 50 based on the two-dimensional slice image group 210 of the subject (see FIG. 5). The estimated lesion space 75 also includes a neighboring space adjacent to the bone of the subject.

[0031] Further, the image processing apparatus 30 obtains, as a region of interest, the product space 77 (hereinafter also referred to as the diseased bone region 77) of at least the surface-containing portion among the estimated lesion space 75 and the three-dimensional bone model 330. For example, the image processing apparatus 30 obtains the product space (hereinafter also referred to as the curved surface region 79) of the estimated lesion space 75 and the surface of the three-dimensional bone model 330 as the diseased bone region 77. Note that the curved surface region 79 is also referred to as a lesion inclusion region (on the bone surface) or a specific curved surface region, etc.

[0032] Then, the image processing apparatus 30 displays, on the display unit 35b, the region of interest (diseased bone region 77) of the three-dimensional bone model 330 in a manner different from that of the portion other than the diseased bone region 77. For example, the image processing apparatus 30 displays, on the display unit 35b, the region of interest (such as the curved surface region 79) on the surface of the three-dimensional bone model in a manner different from that of the region other than the region of interest (for example, a different display color).

[0033] As shown in FIG. 1, the image processing apparatus 30 includes a controller 31, a storage unit 32, and an operation unit 35.

[0034] The controller 31 is a control device built in the image processing apparatus 30 and controls the operation of the image processing apparatus 30.

[0035] The controller 31 is configured as a computer system including one or more hardware processors (e.g., a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit)). The controller 31 realizes various processes by executing a predetermined software program (hereinafter also simply referred to as a program) stored in a storage unit (non-volatile storage units such as a ROM and / or a hard disk) 32 in a CPU or the like. Note that the program (specifically, a program module group) may be recorded on a portable recording medium such as a USB memory, read from the recording medium, and installed in the image processing apparatus 30. Alternatively, the program may be downloaded via a communication network or the like and installed in the image processing apparatus 30.

[0036] The controller 31 executes, for example, processes related to an inference stage in machine learning. Specifically, inference processing is executed for each of a plurality of two-dimensional slice images 220 (240) acquired for a certain subject using a learned model 410 (trained model 420) (see FIG. 2) learned by machine learning. By this inference processing, an estimated lesion region 71 is obtained and an estimated lesion space 75 (see FIG. 5) is formed. The controller 31 also executes output processing related to a lesion site (such as display processing of an image including a fracture site) (see FIG. 6 and the like). Specifically, the controller 31 obtains, as a target region, a product space (lesion bone region) 77 of at least a portion including the surface of the estimated lesion space 75 and the three-dimensional bone model 330. Then, the controller 31 displays, on the display unit 35b, the target region in a manner different from that of the region other than the target region among the three-dimensional bone models. In other words, the controller 31 performs a process different from that for the region other than the target region on the target region.

[0037] The storage unit 32 is composed of a storage device such as a hard disk drive (HDD) and / or a solid state drive (SSD). The storage unit 32 stores a plurality of two-dimensional slice images 220, a learned model 420 (including learning parameters), and the like.

[0038] The operation unit 35 includes an operation input unit 35a that receives operation inputs to the image processing apparatus 30, and a display unit 35b that performs display output of various information. For example, the operation input unit 35a (also referred to as a reception unit) receives a designation operation of the two-dimensional slice image group 210 to be processed. Further, the display unit 35b three-dimensionally displays a fracture site 50 and the like (as a diseased bone region 77) using a three-dimensional bone model 330 based on the inference processing result using the learned model 420. Thereby, the fracture site 50 and the like are clearly shown. As the operation input unit 35a, a mouse, a keyboard, or the like is used, and as the display unit 35b, a display (such as a liquid crystal display) is used. Further, a touch panel that also functions as a part of the operation input unit 35a and also functions as a part of the display unit 35b may be provided.

[0039] Note that this image processing apparatus 30 is also referred to as a medical image processing apparatus, and the image processing system 10 is also referred to as a medical image processing system.

[0040] <1-2. Details of the processing> FIG. 3 is a flowchart showing the processing of the image processing apparatus 30 (specifically, the controller 31 (FIG. 1)). In FIG. 3, processing including inference processing (step S22) using the learned model 420 by machine learning is shown.

[0041] First, in step S21, the image processing apparatus 30 obtains a three-dimensional bone model 330 based on the two-dimensional slice image group 210 (see FIG. 4). FIG. 4 is a diagram conceptually showing that the three-dimensional bone model 330 is generated based on the two-dimensional slice image group 210 (a plurality of two-dimensional slice images 220) regarding the subject.

[0042] Specifically, the image processing apparatus 30 extracts, from among the two-dimensional slice images 220, a set of pixels (picture elements) whose index values indicating bone-likeness are within a predetermined range as a bone region (two-dimensional region). As the index value indicating bone-likeness, for example, the CT value (X-ray absorption value of the subject) of each pixel in the CT image (two-dimensional slice image 220) can be used. More specifically, a set of pixels having values within a predetermined range (for example, +100 to +1900) among the CT values (for example, -1000 to +4000) in the CT image may be extracted as the bone region. Each of these pixels has a predetermined thickness (length in the normal direction of the slice image) and is also expressed as a voxel.

[0043] Then, by stacking the bone regions in the plurality of two-dimensional slice images 220 (in other words, integrating the voxels of each slice image), a three-dimensional bone model 330 is generated. Here, the three-dimensional bone model 330 is generated as a model that includes not only the surface of the bone but also the inside of the bone. In other words, the three-dimensional bone model 330 is formed as a solid model.

[0044] Next, in step S22, as shown in FIG. 5, the image processing apparatus 30 estimates (obtains) an estimated lesion space 75 based on the two-dimensional slice image group 210. Note that FIG. 5 is a conceptual diagram showing that a three-dimensional bone model is generated based on a two-dimensional slice image group regarding a subject.

[0045] Specifically, the image processing apparatus 30 first uses the learned model 420 to estimate, in two or more two-dimensional slice images 220 (also referred to as 250) among the two-dimensional slice image group 210 of the subject, a two-dimensional region including the fracture site 50 in each of the two-dimensional slice images as the estimated lesion region 71 (see the upper half of FIG. 5). The learned model 420 is a model that takes a two-dimensional slice image as input and outputs an estimated lesion region 71 including a lesion site or the like. When the learned model 420 determines that a lesion site (fracture site 50) exists in the input two-dimensional slice image 220, it outputs an estimated lesion region 71 including the fracture site 50 and the like. Next, the image processing apparatus 30 obtains an estimated lesion space 75 based on the estimated lesion regions 71 of the two or more two-dimensional slice images 220 (250) (see the lower half of FIG. 5). Specifically, the estimated lesion space 75 is obtained by stacking the estimated lesion regions 71 extracted from two or more two-dimensional slice images.

[0046] The estimated lesion region 71 is a region including the fracture site 50 (lesion site), and the estimated lesion space 75 is estimated as a space including the fracture site 50. Also, considering that the estimated lesion region 71 may also include a neighboring region adjacent to the bone of the subject, the estimated lesion space 75 is also expressed as a space including a neighboring space adjacent to the bone of the subject. In FIG. 5, the estimated lesion space 75 is illustrated as having a rectangular parallelepiped shape (in other words, each estimated lesion region 71 has the same size and shape as each other), but it is not limited thereto, and the estimated lesion space 75 may have other shapes. For example, the estimated lesion space 75 may be configured as a stack of a plurality of estimated lesion regions 71 having different sizes and / or shapes from each other.

[0047] Furthermore, in steps S23 and S24, the image processing apparatus 30 three-dimensionally displays the position of the lesion (such as the position of the fracture site) in the three-dimensional bone model 330 of the subject.

[0048] Specifically, first, in step S23, a diseased bone region 77 (see FIG. 6), which is a product space of at least a part including the surface of the estimated lesion space 75 and the three-dimensional bone model 330, is obtained as a region of interest.

[0049] The diseased bone region 77 is generated, for example, as a product space 79 (curved surface region 79) of the estimated lesion space 75 and the surface of the three-dimensional bone model 330. However, it is not limited thereto. As will be described later, the diseased bone region 77 may be generated as a product space 78 (see FIGS. 14 and 15) of the estimated lesion space 75 and the three-dimensional bone model 330 (a model including not only the surface of the bone but also the inside of the bone). Note that the curved surface region 79 is equivalent to the surface of the product space 78. Also, the curved surface region 79 is also expressed as the surface of the diseased bone region 77.

[0050] Then, in step S24, among the three-dimensional bone model 330, the diseased bone region 77 is displayed on the display unit 35b in a manner different from that of the part other than the diseased bone region (see FIG. 6). For example, among the surfaces of the three-dimensional bone model 330, the curved surface region 79 (region of interest) is displayed on the display unit 35b in a manner different from that of the region other than the curved surface region 79 (see FIG. 6). More specifically, among the surfaces of the three-dimensional bone model 330, the region 332 other than the curved surface region 79 is displayed in a basic color (such as white), while the curved surface region 79 is displayed in a specific color other than the basic color (such as red). Note that FIG. 6 is a diagram showing a display example of the three-dimensional bone model 330 with the display of the curved surface region 79. In FIG. 6, for convenience of illustration, the curved surface region 79 is shown with hatching. Also, the three-dimensional bone model 330 and the curved surface region 79 are preferably subjected to shading processing for realizing a three-dimensional representation. At that time, it is preferable that the specific color of the curved surface region 79 and the basic color of the region other than the curved surface region 79 are different from each other even when shading processing is performed. For example, among the surfaces of the three-dimensional bone model 330, the region other than the curved surface region 79 may be colored with a gradation color (such as a gray scale) that changes from white or the like, and the curved surface region 79 may be colored with a gradation color that changes from red or the like.

[0051] Note that the three-dimensional bone model 330 (stereoscopic surface model of bone) is appropriately rotated, moved (translated), and scaled (enlarged and / or reduced) according to user operations. In other words, the user can view the three-dimensional bone model 330 (stereoscopic surface model of bone) displayed on the display surface of the display unit 35b from a desired viewpoint and at a desired size. Therefore, the user can view the three-dimensional bone model 330 from various viewpoints and confirm the position of the diseased bone region 77 (such as the curved surface region 79) indicated in a specific color (and thus the position of the fracture site 50).

[0052] Here, a first comparative example (see FIG. 16) will be described. As shown in FIG. 16, in the first comparative example, the estimated lesion space 75 is directly superimposed and displayed on the three-dimensional bone model 330 of the subject. Specifically, the estimated lesion space 75 is represented by a solid model (here, a solid model having a rectangular parallelepiped shape), and the solid model is displayed in a state of being combined with the three-dimensional bone model 330.

[0053] However, the estimated lesion space 75 formed by laminating a relatively wide estimated lesion region 71 (a region including not only the fracture site 50 but also the adjacent vicinity region (outer region of the bone) adjacent to the bone) has a portion protruding outward from the surface of the three-dimensional bone model 330. In other words, a part of the solid model indicating the estimated lesion space 75 protrudes outward from the surface of the three-dimensional bone model 330.

[0054] When such an estimated lesion space 75 is directly superimposed on the three-dimensional bone model 330 of the subject as in this comparative example, as shown in FIG. 16, the surface of the three-dimensional bone model 330 (especially the bone surface near the fracture site 50) is covered by the estimated lesion space 75 and becomes invisible. Therefore, it is difficult to visually recognize the fracture site 50. That is, it becomes difficult to grasp the fracture position and the like in the three-dimensional bone model 330.

[0055] In contrast, in the above embodiment, a lesion bone region 77 (for example, a curved surface region 79), which is a product space of at least a portion including the surface of the estimated lesion space 75 and the three-dimensional bone model 330, is obtained. The lesion bone region 77 does not include a neighboring space (an external space adjacent to the bone) adjacent to the bone of the subject. And only the lesion bone region 77, which is a product space of at least a portion including the surface of the estimated lesion space 75 and the three-dimensional bone model 330 among the estimated lesion space 75, is displayed, and the neighboring space (the external space of the bone) adjacent to the bone of the subject is not visualized and is not displayed as a solid model or the like (see FIG. 6). Therefore, it is possible to avoid the surface of the three-dimensional bone model 330 (especially the surface near the fracture site 50, etc.) from being hidden. As a result, it becomes easier for the user to visually recognize the state near the fracture site 50 (specifically, the fracture space generated by the fracture, etc.) and both sides thereof (both sides on the bone surface).

[0056] Furthermore, among the three-dimensional bone model 330, the lesion bone region 77, which is a product space with the estimated lesion space 75, is displayed in a different manner (for example, a different display color) from the region other than the lesion bone region 77 (see FIG. 6). For example, among the surfaces of the three-dimensional bone model 330, the curved surface region 79, which is a product space with the estimated lesion space 75, is displayed in a different manner from the region other than the curved surface region 79. In particular, as shown in the middle and bottom partial enlarged views of FIG. 6, the curved surface region 79 including the bone surfaces on both sides of the fracture site 50 is displayed in a different manner (displayed in a different color) from other parts.

[0057] Therefore, it is possible for the user to easily visually recognize the position of the lesion bone region 77 in the three-dimensional bone model 330, and more specifically, to easily visually recognize the position of the curved surface region 79 on the surface of the three-dimensional bone model 330. That is, a display that is easy to view is realized.

[0058] In this way, even when a relatively wide space including the fracture site 50 is estimated as the estimated lesion space 75, it is possible to display the fracture site 50 in a three-dimensional manner that is easy to view.

[0059] <1-3. Curved surface region 79> FIG. 14 is a diagram conceptually showing the curved surface region 79. In FIG. 14, the three-dimensional bone model 330 is schematically represented by a cylindrical member, and the estimated lesion space 75 is schematically represented by a rectangular parallelepiped member. As shown in FIG. 14, the curved surface region 79 is directly (and explicitly) obtained as the product space of the estimated lesion space 75 and the surface of the three-dimensional bone model 330, for example, based on the estimated lesion space 75 and the three-dimensional bone model 330 (see the large white arrow).

[0060] Here, the curved surface region 79, which is the product space (79) of the estimated lesion space 75 and the surface of the three-dimensional bone model 330, is equivalent to the surface region of the three-dimensional bone model 330 in the product space 78 of the estimated lesion space 75 and the three-dimensional bone model 330 (including not only the surface of the bone but also models including the inside of the bone). Here, the product space 78 is a space including not only the surface of the bone but also the inside of the bone, and in FIG. 14, it is shown as a partial cylindrical portion (the overlapping space with the estimated lesion space 75) of the three-dimensional bone model 330.

[0061] Utilizing such characteristics and the like, the curved surface region 79, which is the product space of the estimated lesion space 75 and the surface of the three-dimensional bone model 330, may be indirectly (or as a result) obtained based on the estimated lesion space 75 and the three-dimensional bone model 330 (via the product space 78, etc.).

[0062] For example, the product space 78 of the estimated lesion space 75 and the three-dimensional bone model 330 (including not only the surface of the bone but also the inside of the bone) may be obtained first (see the small white arrow in FIG. 14). Further, a curved surface that is also the surface of the three-dimensional bone model 330 in the product space 78 (for example, the further product space of the surface of the three-dimensional bone model 330 and the product space 78) may be obtained as the curved surface region 79. And among the surfaces of the three-dimensional bone model 330, the curved surface region 79 may be displayed in a different manner (such as a different display color) from the region other than the curved surface region 79 (the region of interest).

[0063] Alternatively, after the product space 78 is obtained, the surface of the product space 78 and the surface of the three-dimensional bone model 330 may be superimposed and displayed (overlay display) without explicitly obtaining a further product space (79) between the surface of the product space 78 and the surface of the three-dimensional bone model 330. At this time, the surface of the product space 78 may be displayed in a display color (e.g., red, etc.) different from the basic display color (e.g., white, etc.) of the surface of the three-dimensional bone model 330. By such an overlay display or the like, a portion (a portion other than the surface of the three-dimensional bone model 330) corresponding to the inside of the bone among the surfaces of the product space 78 is hidden under the surface of the three-dimensional bone model 330. Examples of such a portion hidden under the surface of the three-dimensional bone model 330 include a boundary surface (end face 78d of the cylindrical product space 78) between the bone portion adjacent to the estimated lesion space 75 and the surface of the product space 78. As a result, only the surface (the surface region of the three-dimensional bone model 330 in the product space 78) that is also the surface of the three-dimensional bone model 330 in the product space 78 is displayed as the curved surface region 79 in a different manner (e.g., a different display color, etc.) from the region 332 other than the curved surface region 79. In this way, the curved surface region 79 may be displayed in a different manner (e.g., a different display color, etc.) from the region other than the curved surface region 79.

[0064] Also, the curved surface region 79 (the region of interest) does not need to be explicitly separated from the three-dimensional bone model 330 (a model including not only the surface of the bone but also the inside of the bone) or the product space 78, etc. The curved surface region 79 may be displayed as the region of interest in a different manner (e.g., a different display color, etc.) from the region other than the curved surface region 79 (the region of interest) while the curved surface region 79 remains a part of the three-dimensional bone model 330 or the product space 78, etc.

[0065] <1-4. Complete fracture and incomplete fracture> In addition, fractures are classified into "complete fractures" and "incomplete fractures (also referred to as incomplete fractures)". A "complete fracture" (see Figure 7) is a fracture in which the bone is completely broken. On the other hand, an "incomplete fracture" (see Figure 8) is a fracture in which the bone is not completely broken, such as a state where so-called "cracks" have occurred. Note that Figure 7 is a partial enlarged view of the three-dimensional bone model 330 near the fracture site with a complete fracture, and Figure 8 is a partial enlarged view of the three-dimensional bone model 330 near the fracture site with an incomplete fracture. In Figures 7 and 8, the fracture states are shown schematically.

[0066] As shown in Figure 7, in the case of a complete fracture, a gap 55 occurs at the fracture site 50 due to the breakage of the bone. In this case, the curved surface region 79 has a curved surface region (partial region) 79a existing on both sides (left and right sides) of the gap 55 on the surface of the broken bone, and a curved surface region (partial region) 79c existing on the fracture surface of the broken bone (the part that was inside the bone before the fracture). The parts on both sides of the gap 55 on the surface of the bone are also expressed as the parts near the fracture site 50.

[0067] Figure 8 shows a fracture (incomplete fracture) with a crack 54. In this case, the curved surface region 79 has a curved surface region (partial region) 79a existing in the vicinity of both the left and right sides of the crack 54 (fracture site 50) on the surface of the bone.

[0068] The product space of the estimated lesion space 75 and the three-dimensional bone model 330 (and thus the product space of the estimated lesion space 75 and the surface of the three-dimensional bone model 330) does not include the neighboring space adjacent to the subject's bone (the adjacent external space of the bone). That is, the curved surface region 79 does not include the adjacent external space of the bone. And in the neighboring space adjacent to the subject's bone (the adjacent external space of the bone), a solid model or the like indicating the estimated lesion space 75 is not displayed. Therefore, it is possible to avoid the surface of the three-dimensional bone model 330 (especially the state near the fracture site 50 and its both sides) from being hidden. Thus, the user can also grasp the surface state of the bone near the fracture site 50 (the presence or absence of breakage and the state of breakage, as well as the presence or absence of "cracks" and the state of "cracks", etc.).

[0069] Also, on the surface of the three-dimensional bone model 330, these curved surface regions 79 (79a, 79c) are displayed in a different manner (such as coloring display with a specific color) from the region 332 other than the curved surface region 79. According to such a three-dimensional bone model 330, the user can easily grasp the position of the fracture site 50 near the waist of the subject.

[0070] For example, in the case of an incomplete fracture as shown in FIG. 8, on the surface of the three-dimensional bone model 330, the curved surface region 79 (79a) is displayed in a different manner (such as coloring display with a specific color) from the region 332 other than the curved surface region 79. Therefore, the user can easily grasp the position of the fracture site 50 of the subject. Also, as described above, since the curved surface region 79 is the surface of the three-dimensional bone model 330 (excluding the adjacent external space of the bone), the user can also grasp the surface state of the bone near the fracture site 50 (such as the presence or absence of "cracks" and the state of "cracks").

[0071] Also, in the case of a complete fracture as shown in FIG. 7, not the gap 55 itself caused by the fracture, but both sides of the fractured bone and the fracture surface are colored with a specific color as the curved surface region (region of interest) 79 (79a, 79c) and displayed separately from other parts. Therefore, the user can easily grasp the position of the fracture site 50 of the subject. Also, as described above, since the curved surface region 79 is the surface of the three-dimensional bone model 330 (excluding the adjacent external space of the bone), the user can also grasp the surface state of the bone near the fracture site 50 (such as the presence or absence of fractures and the state of fractures). In particular, the presence or absence of the gap 55 (the presence or absence of fractures), as well as the fracture surfaces on both sides of the gap 55 and the bone surfaces on both sides of the gap 55, can be directly visually recognized. Also, instead of the gap 55 itself being colored, etc., the periphery of the gap 55 (specifically, the fracture surface of the bone and both sides or one side (near the fracture site) of the gap 55 on the bone surface) is colored, etc., so that the state near the fracture site 50 is easily visible. In particular, even when the gap 55 itself of the fracture site 50 is small (narrow), etc., a relatively wide range (a range wider than the width of the gap 55 itself) extending over the vicinity of both sides (or one side) of the gap 55 is colored, so that the user can easily visually recognize the fracture site 50.

[0072] <1-5. Product space 78> In the above, the product space 79 (curved surface region 79) of the estimated lesion space 75 and the surface of the three-dimensional bone model 330 is obtained as the diseased bone region 77, and the mode in which the curved surface region 79 is displayed in a different mode from the region other than the curved surface region 79 on the surface of the three-dimensional bone model 330 has been mainly described. However, the present invention is not limited to this.

[0073] For example, as shown in FIG. 15, the product space 78 of the estimated lesion space 75 and the three-dimensional bone model 330 (a model including not only the surface of the bone but also the inside of the bone) may be obtained as the diseased bone region 77. In other words, the same idea as above may be applied to each voxel inside the three-dimensional bone model 330.

[0074] Specifically, in step S23, the product space 78 (see FIG. 15) of the estimated lesion space 75 and the three-dimensional bone model 330 (specifically, its bone part) may be obtained as the diseased bone region 77. Note that in FIG. 15 (especially the middle part thereof), the product space 78 and the like of the estimated lesion space 75 and the three-dimensional bone model 330 are schematically shown.

[0075] Then, in step S24, this product space 78 may be displayed on the display unit in a different mode from the part other than the product space 78 (diseased bone region) in the three-dimensional bone model 330. More specifically, the voxels constituting the product space 78 and the voxels constituting the part other than the product space 78 (diseased bone region) may be displayed in different modes (with different display colors, etc.). In other words, not only the surface (each voxel) of the three-dimensional bone model 330 but also the inside (each voxel) of the three-dimensional bone model 330 may be subjected to the same display process as the surface of the three-dimensional bone model 330.

[0076] This also enables the realization of a display similar to that in FIG. 6 and the like. For example, the surface of the portion of the three-dimensional bone model 330 corresponding to the product space 78 (lesion bone region 77) (in other words, the curved surface region 79) may be displayed in a manner different from the surface region (normal region) other than the product space 78 (lesion bone region 77), similar to the above-described embodiments and the like.

[0077] Also, in the case of performing a display with a cross-section of the three-dimensional bone model 330, etc., the inside of the portion of the cross-section corresponding to the product space 78 (the voxels inside the three-dimensional bone model 330) may also be displayed in a different manner (different display colors, etc.) from the portion other than the product space 78 (lesion bone region 77). In the lowermost part of FIG. 15, the product space 78 is cut by a certain cut surface 78c, and the voxels inside the product space 78 (three-dimensional bone model 330) are exposed at the cut surface 78c. As shown in the lowermost part of FIG. 15, the inside of the product space 78 appearing on the cut surface 78c may also be displayed in a manner different from the portion other than the lesion bone region 77 (product space 78) of the three-dimensional bone model 330. Note that the inside of the product space 78 may be displayed in the same manner as the surface of the product space 78, or may be displayed in a manner different from the surface of the product space 78.

[0078] Also, the product space 78 does not need to be explicitly extracted. To put it simply, step S23 (see FIG. 3) may not be executed. For example, when the controller 31 obtains the estimated lesion space 75 (step S22), for each voxel in a predetermined space that includes all of the three-dimensional bone model 330 (or each voxel on the surface and inside of the three-dimensional bone model 330), a flag indicating whether it is a voxel within the estimated lesion space 75 is further assigned. Specifically, a "1" is assigned to the voxels within the estimated lesion space 75, and a "0" is assigned to the voxels outside the estimated lesion space 75. Then, when the three-dimensional bone model 330 is displayed (step S24), among the voxels on the surface and inside of the three-dimensional bone model 330, the voxels having the flag value "1" may be displayed in a display color different from that of the voxels having the flag value "0". In this case, among the voxels on the surface and inside of the three-dimensional bone model 330, the voxels having the flag value "1" correspond to the voxels of the product space 78 (specifically, its surface and inside), and the voxels having the flag value "0" correspond to the voxels of the portion other than the product space 78. In such a manner, the product space 78 (lesion bone region 77) may be displayed in a manner different from that of the portion other than the product space 78 in the three-dimensional bone model 330.

[0079] <2. Second Embodiment> The second embodiment is a modification of the first embodiment. Hereinafter, the description will focus on the differences from the first embodiment.

[0080] In the above first embodiment, the estimated lesion space 75 is formed based on one two-dimensional slice image group 210 regarding a single reference axis direction, but it is not limited to this. For example, the estimated lesion space 75 may be formed based on a plurality of two-dimensional slice image groups 210 regarding different reference axis directions. In the second embodiment, such a manner will be described.

[0081] The plurality of slice image groups 210 are obtained for each of a plurality of different reference axes (the normal direction of the cross section) with respect to the same subject. In other words, the slice image groups 210 (a plurality of two-dimensional slice images) are further photographed after changing the direction of the cross section. As a result, a plurality of slice image groups 210 are imaged (generated).

[0082] Each slice image group 210 has a plurality of slice images obtained by slicing the subject with a cross section having a unique normal direction for each image group. Note that each slice image group 210 is also referred to as a (the i-th) slice image group Gi (where i = 1,..., N; N is the number of slice image groups and is a natural number of 2 or more).

[0083] Each slice image group 210 is imaged by slicing the subject with a cross section in a direction different from the cross sections of other slice image groups (for example, each cross section such as an axial cross section, a coronal cross section, a sagittal cross section, etc.) (see FIG. 10).

[0084] FIG. 10 is a diagram showing various types of cross sections of an axial cross section, a coronal cross section, and a sagittal cross section. The axial cross section is a cross section orthogonal to the body axis (also referred to as an axial section or a transverse section). The coronal cross section is a cross section that cuts across (a cross section that divides the body, etc. into a front side and a rear side) (also referred to as a coronal section or a coronal plane). The sagittal cross section is a longitudinal cross section (a cross section that divides the body, etc. into a left side and a right side) (also referred to as a sagittal section or a sagittal plane). In addition, as types of cross sections, there are also oblique cross sections (tilted cross sections) (oblique sections), etc.

[0085] One slice image group 210 is composed of a plurality of two-dimensional slice images 220 sliced by an axial cross section (a plurality of two-dimensional slice images 220 having the normal direction of the axial cross section as the reference axis direction) (see FIG. 4).

[0086] Another slice image group 210 is composed of a plurality of two-dimensional slice images 220 sliced in the coronal plane (a plurality of two-dimensional slice images 220 with the normal direction of the coronal plane as the reference axis direction). Yet another slice image group 210 is composed of a plurality of two-dimensional slice images 220 sliced in the sagittal plane (a plurality of two-dimensional slice images 220 with the normal direction of the sagittal plane as the reference axis direction). Further, still another slice image group 210 is composed of a plurality of two-dimensional slice images 220 sliced in an oblique plane ((a plane) inclined at an arbitrary angle) (oblique section) (see FIG. 11 etc.).

[0087] Note that the plurality of slice image groups 210 may be obtained by imaging slice image groups in respective directions (for example, each of the nine directions) with respect to mutually different reference axes as described above. However, it is not limited to this, and the plurality of slice image groups may be obtained by imaging one slice image group in one direction and performing image conversion processing on the one slice image group to generate slice image groups in other directions (for example, eight slice image groups with the other eight directions as the reference axis directions respectively).

[0088] FIG. 11 is a conceptual diagram for explaining a lesion candidate space 85 (described later) and an integrated lesion candidate space 87 (described later) in each of a plurality of slice image groups. A rectangular parallelepiped 80 in FIG. 11 conceptually shows a slice image group 210 (a stack of a plurality of two-dimensional slice images 220). The nine large rectangular parallelepipeds 80 on the left half of FIG. 11 respectively correspond to nine slice image groups 210. Also, the thick lines drawn in a parallelogram shape in each of the nine large rectangular parallelepipeds 80 indicate a cross section (slice cross section) in each slice image group 210. Among the nine rectangular parallelepipeds 80 in the middle row (arranged in the left-right direction) on the left half of FIG. 11, three rectangular parallelepipeds conceptually show three slice image groups 210 (in order from the left, a slice image group 210 by a coronal cross section, a slice image group 210 by an axial cross section, and a slice image group 210 by a sagittal cross section). Also, in the upper and lower rows on the left half of FIG. 11, a total of six slice image groups 210 by oblique cross sections (three each) are conceptually shown. The six slice image groups 210 are slice image groups sliced by six different oblique cross sections (with the normal directions of the six different oblique cross sections as the reference axis directions respectively). Note that when each of the nine two-dimensional slice image groups 210 is imaged, strictly speaking, the orientations of the respective rectangular parallelepipeds 80 are different from each other depending on the direction of the reference axis of each two-dimensional slice image group 210. Here, for the main purpose of showing the existence of the two-dimensional slice image group 210, the two-dimensional slice image group 210 is conceptually represented by the rectangular parallelepiped 80.

[0089] In the second embodiment, a lesion candidate space 85 in each of a plurality of slice image groups 210 and an integrated lesion candidate space 87 obtained by integrating the plurality of lesion candidate spaces 85 are formed, and further, an integrated lesion space 89 (see the right end of FIG. 11) is extracted from the integrated lesion candidate space 87. Then, the integrated lesion space 89 is obtained as the estimated lesion space 75 (also referred to as 75B) in the second embodiment.

[0090] In the second embodiment, the same processing as in the first embodiment (see FIG. 3) is executed. However, the processing in step S22 is different from that in the first embodiment. FIG. 9 is a flowchart showing the processing in step S22 of the second embodiment. Hereinafter, the processing of step S22 (also referred to as S22B) according to the second embodiment will be described with reference to FIG. 9.

[0091] First, in step S31, the controller 31 acquires (inputs) the i-th two-dimensional slice image group Gi (the initial value of the value i is "1" here) in response to a designation operation by the operation user. Specifically, the storage folder and file name, etc., of the i-th two-dimensional slice image group Gi (210) are designated by the operation user, and the controller 31 reads the data of the i-th slice image group Gi based on the designated content.

[0092] In the next step S32, the controller 31 executes an inference process by the learned model 420 for each of the plurality of two-dimensional slice images 240 constituting the i-th two-dimensional slice image group Gi (simply referred to as the i-th slice image group). Specifically, the region estimated as the lesion region (estimated lesion region 71) in the i-th slice image group Gi (210) is obtained by the learned model 420 (learner).

[0093] Then, in step S33, a lesion candidate space 85 is formed based on the estimated lesion regions 71 detected in each two-dimensional slice image 220 of the i-th slice image group Gi. For example, the lesion candidate space 85 is formed by stacking the estimated lesion regions 71 detected in each two-dimensional slice image 220 of the i-th slice image group Gi. The lesion candidate space 85 is a candidate space for lesions (specifically, a candidate space for a space including lesions), and is a space estimated to include lesions. The lesion candidate space 85 is also expressed as a space corresponding to the estimated lesion space 75 (also referred to as 75A) in the first embodiment.

[0094] In the next step S34, a branch process is executed according to whether the processing has been completed for all the slice image groups Gi. If the processing has not been completed, the value i is incremented (step S35), and the process returns to step S31 to execute the processes of steps S31 to S33. On the other hand, if the processing has been completed for all the slice image groups Gi (when the lesion candidate spaces 85 have been formed for all of the N slice image groups Gi (G1 to GN)), the process proceeds to step S36.

[0095] In step S36, an integrated lesion candidate space 87 obtained by integrating the N lesion candidate spaces 85 is formed (see FIG. 11). Specifically, the integrated lesion candidate space 87 is formed as the logical sum space of the N lesion candidate spaces 85. More specifically, the integrated lesion candidate space 87 is formed by integrating the lesion candidate spaces 85 of the plurality of slice image groups Gi while aligning their respective corresponding positions. In FIG. 11, the lesion candidate spaces 85 (nine lesion candidate spaces 85) for each of the nine slice image groups G1 to G9 are schematically shown, and the integrated lesion candidate space 87 obtained by integrating the nine lesion candidate spaces 85 is schematically shown.

[0096] Then, in step S37, an integrated lesion space 89 is specified based on the integrated lesion candidate space 87. Specifically, the controller 31 finally determines, as the lesion region, the voxels belonging to more than a predetermined number (for example, five) of the lesion candidate spaces 85 among the plurality of voxels in the integrated lesion candidate space 87. On the right end side of FIG. 11, the integrated lesion space 89 (the finally estimated (determined) lesion inclusion space) is schematically shown. The integrated lesion space 89 is an aggregate of voxels belonging to more than a predetermined number (for example, five) of the lesion candidate spaces 85 among the voxels in the integrated lesion candidate space 87.

[0097] More specifically, among a plurality of voxels in a three-dimensional model related to a subject (three-dimensional volume data (a model including bones, internal organs, etc.) based on a plurality of two-dimensional slice images 220), voxels within the lesion candidate space 85 are assigned an evaluation value of "1", and voxels outside the lesion candidate space 85 are assigned an evaluation value of "0" (see FIG. 11). Then, when integrating the plurality of slice image groups Gi, the evaluation values are added for each corresponding voxel. The evaluation value for each voxel after addition represents how many slice image groups Gi the voxel belongs to in the lesion candidate space 85. For example, if the evaluation value of a certain voxel is "9", it means that the certain voxel belongs to all (9) of the lesion candidate spaces 85 of 9 slice image groups Gi. Also, if the evaluation value of a certain voxel is "7", it means that the certain voxel belongs to the lesion candidate space 85 of 7 slice image groups Gi. Further, if the evaluation value of a certain voxel is "0", it means that the certain voxel does not belong to any of the lesion candidate spaces 85 of the 9 slice image groups Gi.

[0098] Here, the integrated lesion space 89 may be defined as an aggregate including voxels belonging to at least one lesion candidate space 85. However, voxels belonging to only a small number of lesion candidate spaces 85 are likely not to be voxels related to the lesion region in reality. Therefore, it is preferable that a voxel is determined as a lesion region on the condition that it is also detected as an estimated lesion region 71 in the lesion candidate spaces 85 of many other slice image groups Gi. In other words, it is preferable that a voxel is determined as a lesion region on the condition that it is detected as an estimated lesion region 71 when viewed from other angles. Thus, here, a space composed of voxels belonging to more than a predetermined number (for example, 5) of lesion candidate spaces 85 is estimated (determined) as the final lesion space (integrated lesion space 89).

[0099] In the second embodiment, the integrated lesion space 89 extracted in this way is determined as the estimated lesion space 75 (75B).

[0100] Thereafter, in the same manner as in the first embodiment, in steps S23 and S24 (see FIG. 3), the image processing apparatus 30 three-dimensionally displays the position of a lesion (such as the position of a fracture site) and the like in the three-dimensional bone model 330 of the subject.

[0101] Specifically, first, in step S23, a lesion bone region 77, which is a product space of a portion including at least the surface of the three-dimensional bone model 330 and the estimated lesion space 75 (here, the integrated lesion space 89), is obtained as a region of interest. Then, in step S24, among the surfaces of the three-dimensional bone model 330, the region of interest (lesion bone region 77) is displayed on the display unit 35b in a manner different from the regions other than the region of interest.

[0102] FIG. 12 is a diagram showing an example of the display of the three-dimensional bone model 330 in step S24. As can be seen by comparison with the comparative example (the second comparative example) in FIG. 17, various effects can be obtained by the display as shown in FIG. 12.

[0103] In the comparative example (the second comparative example) in FIG. 17, the integrated lesion space 89 (estimated lesion space 75) extracted in step S37 is formed as a solid model, and the integrated lesion space 89 (solid model) is combined with the three-dimensional bone model 330 and displayed. As can be seen by comparison with FIG. 16, the integrated lesion space 89 (the estimated lesion space 75 (75B) according to the second embodiment) in FIG. 17 does not have a rectangular parallelepiped shape like the estimated lesion space 75 (75A) in FIG. 16, but has a complex shape. Therefore, for example, as shown in FIG. 13, the integrated lesion space 89 also has a complex shape in a certain cross section (FIG. 13 is a certain coronal cross section). In other words, the product space (product plane) of the integrated lesion space 89 and the certain coronal cross section has a complex shape. Note that FIG. 13 is a cross-sectional view showing the estimated lesion space 75 and the like cut by a certain coronal cross section. In FIG. 13, the cross-sectional region obtained by cutting the integrated lesion space 89 (estimated lesion space 75) by a certain coronal cross section is shown surrounded by a thick black line.

[0104] As shown in FIG. 17, the estimated lesion space 75 (the integrated lesion space 89) in the second comparative example has a portion protruding outward from the surface of the three-dimensional bone model 330. For example, in the slice image 220 of FIG. 13, a cross-sectional area (the area surrounded by the thick black line) obtained by cutting the integrated lesion space 89 (estimated lesion space 75) with a certain coronal section includes not only the bone area 75a (the area corresponding to the inside and surface of the bone), but also a portion 75c protruding outward from the surface of the bone.

[0105] When such an estimated lesion space 75 (integrated lesion space 89) is directly superimposed on the three-dimensional bone model 330 of the subject as in the second comparative example, as shown in FIG. 17, the surface of the three-dimensional bone model 330 is covered by the estimated lesion space 75 and becomes invisible. For example, although there is actually a gap 55 due to a fracture in the upper part of the ilium (see FIG. 12), in FIG. 17, the fracture site (near the gap 55) is covered by the estimated lesion space 75, and the user cannot see the fracture site. Similarly, the fracture parts existing in the left and right pubic parts and the left and right ischial parts are also covered by the estimated lesion space 75, and the user cannot see the fracture site. Thus, it is difficult to visually recognize the fracture site 50. That is, it is difficult to grasp the fracture position and the like in the three-dimensional bone model 330.

[0106] In contrast, in the second embodiment, a lesion bone region 77, which is a product space of the estimated lesion space 75 (integrated lesion space 89) and a portion including at least the surface of the three-dimensional bone model 330, is displayed in a manner different from a region other than the lesion bone region 77 on the surface of the three-dimensional bone model 330. In particular, only the lesion bone region 77, which is a product space of the estimated lesion space 75 and a portion including at least the surface of the three-dimensional bone model 330, is displayed among the estimated lesion space 75, and a neighboring space (adjacent external space of the bone) adjacent to the bone of the subject is not visualized and is not displayed as a solid model or the like. Therefore, it is possible to avoid the surface of the three-dimensional bone model 330 (particularly, the state of the fracture site 50 and the vicinity of both sides thereof) from being hidden. Specifically, fracture sites such as the upper ilium, fracture sites of the left and right pubic bones, and fracture sites of the left and right ischial bones are displayed without being hidden.

[0107] Furthermore, a lesion bone region 77 (particularly, its surface region), which is a product space with the estimated lesion space 75, among the surface of the three-dimensional bone model 330 is displayed in a manner different from other regions (for example, a different display color). Therefore, the user can easily visually recognize the position of the lesion bone region 77 (particularly, its surface region) among the surface of the three-dimensional bone model 330. That is, a display that is easy to see is realized. Note that the lesion bone region 77 in FIG. 12 may be a product space 78 or a curved surface region 79 as described above.

[0108] In this way, even when a relatively wide space including the fracture site 50 is estimated as the estimated lesion space 75 (integrated lesion space 89), it is possible to display the fracture site 50 in a three-dimensional manner that is easy to see.

[0109] Note that, in the second embodiment, the integrated lesion space 89 is defined as an aggregate or the like that also includes voxels belonging to less than a predetermined number of lesion candidate spaces 85, but is not limited thereto. For example, among a plurality of voxels in the integrated lesion candidate space 87, voxels whose evaluation values are higher than a predetermined level may be determined as the voxels constituting the integrated lesion space 89. As the evaluation value, a value assigned to each voxel according to the reliability calculated by the learned model 420 for the estimated lesion region 71 of each slice image group Gi may be used.

[0110] <3. Modifications, etc.> The embodiments of the present invention have been described above, but the present invention is not limited to the above-described content.

[0111] <Display, etc. according to reliability> For example, in each of the above embodiments, etc., the lesion bone region 77 (product space 78 or product space 79) is displayed in a specific single color, but is not limited thereto.

[0112] Specifically, the display color of each position in the diseased bone region 77, which is the product space of the estimated lesion space 75 and the portion including at least the surface of the three-dimensional bone model 330, may be changed according to the reliability regarding the lesion estimation at each position (the reliability for each position in the estimated lesion space 75). Here, it is assumed that the reliability regarding the estimation that a position is a fracture site (and the vicinity of the fracture site) has been calculated for each voxel in the estimated lesion space 75 in each of the above embodiments. Then, each voxel (or pixel) corresponding to the diseased bone region 77 (such as the curved surface region 79) may be displayed in a color corresponding to the reliability of each voxel (the reliability for each position in the estimated lesion space 75 (and within the diseased bone region 77)). In short, a heat map corresponding to the reliability of each position in the diseased bone region 77 may be generated on the surface of the three-dimensional bone model 330. Also, the color of each position in the diseased bone region 77 may be a color that continuously changes according to the reliability, but is not limited thereto, and may be a specific color for each stage classified into a plurality of stages (the first level to the Nth level) according to the reliability. For example, the first level (reliability 95% or more and 100% or less) may be shown in red, the second level (reliability 80% or more and less than 95%) may be shown in orange, the third level (reliability 50% or more and less than 80%) may be shown in green, and the fourth level (reliability less than 50%) may be shown in yellow.

[0113] According to this, since the display mode of each position in the diseased bone region 77 of the three-dimensional bone model 330 is changed according to the reliability, it is easy for the user to grasp the reliability (the reliability of being a fracture) at each position. Therefore, it is particularly useful when sequentially checking locations considered to be diseased sites (fracture sites) with priorities based on the reliability.

[0114] In addition, stage data may be generated by classifying the diseased bone region 77 into a plurality of stages according to the reliability, and the stage data may be output to different data files (stage data files) for each stage. Specifically, first-level data (first-stage data) indicating the position information of the first-level region in the diseased bone region 77 may be output to the first data file. Also, second-level data (second-stage data) indicating the position information of the second-level region in the diseased bone region 77 may be output to the second data file. Similarly, third-level data indicating the position information of the third-level region in the diseased bone region 77 may be output to the third data file, and fourth-level data indicating the position information of the fourth-level region in the diseased bone region 77 may be output to the fourth data file. These stage data (data corresponding to each stage) are data that can be used when individually displaying the diseased bone region 77 at each stage (level), and are also referred to as display data for the stage-by-stage curved surface region. Further, by using a combination of a plurality of stage data, it is also possible to change the display mode (display color, etc.) of each position in the diseased bone region 77 according to the reliability regarding the lesion estimation at each position. Note that a file having each display data is also referred to as a display data file.

[0115] By generating such a data file, thereafter, in the image processing apparatus 30, it is possible to immediately display the three-dimensional bone model 330 having the diseased bone region 77 (without involving new image processing (for example, steps S21 to S23 in FIG. 3)). Also, it is possible to enhance the affinity with a display device or the like outside the image processing apparatus 30 (particularly, a display device or the like in a system other than the image processing system 10). For example, in another existing system or the like capable of displaying using an external display data file, it is possible to easily perform display using the data file output from the present apparatus 30. In other words, by reading the data file in another system, it is possible to easily realize a display as shown in FIG. 12 in another system.

[0116] <Display according to the type of lesion, etc.> In addition, in each of the above embodiments and the like, the diseased bone region 77 is displayed in a certain display color regardless of the type of lesion, but the present invention is not limited thereto. For example, the display mode of the diseased bone region 77 (the product space of at least the surface of the estimated lesion space 75 and the three-dimensional bone model 330) may be changed according to the type of lesion (type of fracture). Specifically, it may be determined whether the diseased bone region 77 is a region related to a complete fracture or a region related to an incomplete fracture, and based on the determination result, it may be displayed in different modes. More specifically, the diseased bone region 77 related to a complete fracture may be displayed in a specific color (for example, red), and the diseased bone region 77 related to an incomplete fracture may be displayed in a color different from the specific color (for example, blue).

[0117] According to this, since the display mode is changed according to the type of lesion (type of fracture), the user can easily grasp the type of lesion and then visually recognize the lesion location (fracture location).

[0118] In addition, each voxel corresponding to the diseased bone region 77 may also be displayed in a color corresponding to its reliability (the reliability assigned to each voxel). For example, the diseased bone region 77 related to a complete fracture may be displayed in various warm colors (colors corresponding to the reliability of each voxel (each position)), and the diseased bone region 77 related to an incomplete fracture may be displayed in various cool colors (colors corresponding to the reliability of each voxel (each position)).

[0119] Alternatively, a display mode of displaying in a color corresponding to the reliability and a display mode of displaying in a color corresponding to the type of fracture may be selected according to a user operation.

[0120] <Display of small area part> Among the diseased bone region 77 (such as the curved surface region 79, etc.) on the surface of the three-dimensional bone model 330, a small-area suspected part (for example, the suspected part that can hardly be seen as a point in FIGS. 6 and 12) is difficult to see as it is. Therefore, the small-area suspected part may be highlighted. For example, the area of the small-area suspected part may be expanded (for example, by several times) and displayed. Alternatively, the small-area suspected part may be made to blink. Also, the small-area suspected part may be made to blink after being expanded.

[0121] In addition, the highlighting mode of the small-area suspected part and other display modes may be configured to be selectable by the user. When the highlighting mode is selected according to the user's selection operation, the small-area suspected part may be highlighted.

[0122] According to this, it is possible to prevent overlooking the small-area suspected part (and thus overlooking a fracture existing in the suspected part, etc.). Note that a user (such as a doctor) who notices the existence of the small-area suspected part can accurately determine whether there is a fracture, etc. in detail by appropriately magnifying the three-dimensional bone model 330 as needed.

[0123] <Formation of the estimated lesion space 75, etc.> In each of the above embodiments, etc., the estimated lesion space 75 is formed based on the estimated lesion regions 71 in two or more two-dimensional slice images. However, it is not necessary for the estimated lesion space 75 to be actually generated as data obtained by laminating and combining the estimated lesion regions 71 in two or more two-dimensional slice images. It is sufficient for the estimated lesion space 75 to be conceptually formed by the fact that the estimated lesion regions 71 are obtained (extracted) in two or more two-dimensional slice images. In other words, obtaining the estimated lesion regions 71 in two or more two-dimensional slice images 220 itself is equivalent to forming the estimated lesion space 75.

[0124] Also, in each of the above-described embodiments, etc., after the estimated lesion space 75 is once explicitly extracted based on the estimated lesion region 71, the lesion bone region 77 (such as the curved surface region 79), which is the product space of the estimated lesion space 75 and the portion including at least the surface of the three-dimensional bone model 330, is obtained. However, it is not limited to this. For example, the explicit generation process of the estimated lesion space 75 may be omitted.

[0125] Specifically, in each of two or more two-dimensional slice images, the product region (product curve) of the estimated lesion region 71 and the curve corresponding to the bone surface of the three-dimensional bone model 330 may be obtained. The curved surface (laminated curved surface) formed by laminating the product curves in the two or more two-dimensional slice images corresponds to the product space (curved surface region 79) of the estimated lesion space 75 and the surface of the three-dimensional bone model 330. Also by such a method, the product space (curved surface region 79) of the estimated lesion space 75 and the surface of the three-dimensional bone model 330 can be obtained.

[0126] Alternatively, in each of two or more two-dimensional slice images, the product region (product region surface) of the estimated lesion region 71 and the bone cross-section of the three-dimensional bone model 330 may be obtained. The space (laminated space) formed by laminating the product region surfaces in the two or more two-dimensional slice images corresponds to the product space 78 of the estimated lesion space 75 and the three-dimensional bone model 330. Also by such a method, the product space 78 of the estimated lesion space 75 and the three-dimensional bone model 330 can be obtained.

[0127] Note that, as described above, obtaining the estimated lesion region 71 in two or more two-dimensional slice images 220 is equivalent to forming the estimated lesion space 75, and in such a manner, the estimated lesion space 75 is also (substantially) formed.

[0128] <Complementation of the estimated lesion region 71, etc.> In each of the above embodiments and the like, the estimated lesion space 75 is formed based only on the estimated lesion region 71 identified by the estimation process using the learned model 420. However, the present invention is not limited to this, and the estimated lesion space 75 may be formed including regions other than the estimated lesion region 71. For example, when the estimated lesion region 71 is not extracted only in some of the plurality of consecutive slice images Lj (j = 1,..., N), for example, Lk (for example, k = 3, 5), the estimated lesion region 71 may be complemented and generated in the slice image Lk. Specifically, the estimated lesion region 71 (complemented estimated lesion region 71) of the slice image Lk may be generated using the estimated lesion regions 71 of the slice images L(k-1), L(k+1), etc. before and after the slice image Lk. In other words, the estimated lesion region 71 is not limited to the region identified by the estimation process using the learned model 420 and may include other regions. Then, the estimated lesion space 75 may be formed including the complemented estimated lesion region 71 as well.

[0129] <Direct estimation of the estimated lesion space 75 based on three-dimensional volume data, etc.> In each of the above embodiments and the like (step S22 and the like), the estimated lesion region 71 is obtained based on the two-dimensional slice image data using the learned model 420, and the estimated lesion space 75 is obtained based on the estimated lesion region 71. However, the present invention is not limited to this.

[0130] For example, the estimated lesion space 75 may be formed without extracting the estimated lesion region 71. Specifically, the estimated lesion space 75 may be directly obtained from the three-dimensional volume data based on the two-dimensional slice image group 210. Specifically, a pre-generated learned model 420 (also referred to as 420B) that has been machine-learned using teacher data regarding three-dimensional volume data including lesion positions and the like is used. The learned model 420 is a type of learning model that directly extracts the estimated lesion space 75 from the three-dimensional volume data based on the two-dimensional slice image group 210. For example, the learned model 420 is a model that takes three-dimensional volume data as input and outputs an estimated lesion space 75 including lesion sites and the like. Then, three-dimensional volume data of the target person may be input to the learned model 420, and the estimated lesion space 75 may be directly obtained as the output of the learned model 420.

[0131] In addition, when lesion estimation based on the two-dimensional slice image 220 is performed (the estimated lesion region 71 is obtained) as in each of the above embodiments, the processing load related to lesion estimation can be reduced compared to the case where the estimated lesion space 75 is directly estimated based on the three-dimensional volume data.

[0132] <3D bone model 330> Also, in the above embodiments and the like, the three-dimensional bone model 330 is generated as a model (also referred to as a solid model) that includes not only the surface of the bone but also the inside of the bone, but is not limited thereto. The three-dimensional bone model 330 may be generated, for example, as a surface model that shows only the surface of the bone.

[0133] In addition, when the three-dimensional bone model 330 is constructed as a solid model, the product space 78 is formed as a three-dimensional region (space region). On the other hand, when the three-dimensional bone model 330 is constructed as a surface model, the product space 78 is substantially formed as a two-dimensional region (curved surface region). Thus, the product space 78 (lesion bone part region) may be formed as a three-dimensional region or as a two-dimensional region.

[0134] <Others> In addition, in the above-described embodiments and the like, the estimated lesion region 71 (and thus the estimated lesion space 75) in the slice image 220 is detected using machine learning, but it is not limited thereto. For example, the estimated lesion region 71 (and thus the estimated lesion space 75) in the slice image 220 may be detected using other image processing techniques without machine learning or the like.

Explanation of Signs

[0135] 10 Image processing system 20 Slice image generation device 30 Image processing device 210 Two-dimensional slice image group 220 Two-dimensional slice image 330 Three-dimensional bone model 332 Bone surface region (other than the curved surface region 79) 50 Fracture site 55 Gap 71 Estimated lesion region 75, 75A, 75B Estimated lesion space 77 Lesion bone region 78 Product space 79 Curved surface region 85 Lesion candidate space 87 Integrated lesion candidate space 89 Integrated lesion space (estimated lesion space)

Claims

1. An estimated lesion space estimated as a space including a fracture site of the subject based on a two-dimensional slice image group of the subject, the estimated lesion space including a neighboring space adjacent to a bone having the fracture site and protruding outward from the surface of the bone, and a product space of at least a portion including at least the surface of a three-dimensional bone model that is a three-dimensional model of the bone of the subject, and a control unit that displays the lesion bone region on a display unit in a manner different from a portion other than the lesion bone region in the three-dimensional bone model. An image processing apparatus comprising the same.

2. The control unit estimates the estimated lesion space using a learned model learned by machine learning. The image processing apparatus according to claim 1.

3. The control unit estimates, as an estimated lesion region, a two-dimensional region including a fracture site in each of two or more two-dimensional slice images among the two-dimensional slice image group of the subject using a learned model learned by machine learning, and obtains the product space of the estimated lesion space and the surface of the three-dimensional bone model based on the estimated lesion regions of the two or more two-dimensional slice images. The image processing apparatus according to claim 1 or claim 2.

4. The control unit changes a display mode of each position of the lesion bone region, which is a product space of at least a portion including at least the surface of the estimated lesion space and the three-dimensional bone model, according to a reliability regarding lesion estimation at each position. The image processing apparatus according to any one of claims 1 to 3.

5. The control unit generates stage-specific data obtained by classifying the lesion bone region, which is a product space of at least a portion including at least the surface of the estimated lesion space and the three-dimensional bone model, into a plurality of stages according to a reliability regarding lesion estimation, and outputs the stage-specific data to different data files for each stage. The image processing apparatus according to any one of claims 1 to 4.

6. The control unit changes a display mode of the lesion bone region, which is a product space of at least a portion including at least the surface of the estimated lesion space and the three-dimensional bone model, according to a type of lesion. The image processing apparatus according to any one of claims 1 to 5.

7. a) An estimated lesion space estimated as a space including a fracture site of the subject based on a two-dimensional slice image group of the subject, the estimated lesion space including a neighboring space adjacent to the bone having the fracture site and protruding outward from the surface of the bone, and a lesion bone region which is a product space of at least a portion including the surface of a three-dimensional bone model which is a three-dimensional model of the bone of the subject, are displayed on a display unit in a manner different from a portion other than the lesion bone region in the three-dimensional bone model; An image processing method, characterized by comprising the above.

8. a) An estimated lesion space estimated as a space including a fracture site of the subject based on a two-dimensional slice image group of the subject, the estimated lesion space including a neighboring space adjacent to the bone having the fracture site and protruding outward from the surface of the bone, and a lesion bone region which is a product space of at least a portion including the surface of a three-dimensional bone model which is a three-dimensional model of the bone of the subject, are displayed on a display unit in a manner different from a portion other than the lesion bone region in the three-dimensional bone model; A program for causing a computer to execute the above.

Images