Image processing apparatus, image processing method, and recording medium

Abstract

To provide a graphical user interface (GUI) which allows an inner structure of an object including a living body to be easily recognized.SOLUTION: An image processing device comprises: an input reception unit which is configured to receive change of at least one of the first depth and the second depth according to the operation content of a pointing device when the operation position of the pointing device is on a profile image; an inner structure image output unit which is configured to cause a display device to re-display an inner structure image by reconstructing the inner structure image obtained by changing a chromatic parameter within a depth range after the change received by the input reception unit in the same manner as the change mode of the chromatic parameter before the change from the first depth to the second depth after the change on the basis of volume data showing a three-dimensional inner structure of the object; and a profile image output unit which is configured to cause the display device to re-display the profile image by reconstructing the profile image in accordance with the first depth and the second depth after the change received by the input reception unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a recording medium.

Background Art

[0002] An image generation apparatus has been developed that reconstructs an image based on signal data of acoustic waves obtained by measuring a predetermined subject part of a subject.

[0003] For example, research on an image generation apparatus that irradiates a subject such as a living body with light from a light source (such as a laser) and visualizes information inside the subject has been actively promoted in the medical field. As one of such visualization techniques using light, there is Photoacoustic Tomography (PAT: sometimes also referred to as photoacoustic tomography or optoacoustic tomography). In an imaging apparatus using photoacoustic tomography, the irradiated light propagates through the subject and acoustic waves (typically ultrasonic waves) generated from a light-absorbing biological tissue that has absorbed the energy of the diffused light are detected at a plurality of locations around the subject. Then, the obtained signals are mathematically analyzed to visualize information related to the optical characteristic values inside the subject, particularly the absorption coefficient distribution. In recent years, non-clinical research on imaging blood vessel images of small animals using this photoacoustic tomography apparatus and clinical research for applying this principle to image diagnosis such as breast cancer or preoperative planning in the field of plastic surgery have been actively promoted.

[0004] Patent Document 1 discloses a technique that enables estimation of the interface of a subject using image data. Further, Patent Document 2 discloses a technique for improving the discrimination accuracy between superficial blood vessels and body hair included in an internal structure image reconstructed from photoacoustic signals.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] An object of the present invention is to provide a graphical user interface (GUI) that makes it easier to recognize the internal structure of an object including a living body.

Means for Solving the Problems

[0007] According to one aspect of the present invention, Based on volume data showing the three-dimensional internal structure of an object, within a depth range from a first depth to a second depth that is equal to or greater than the first depth and equal to or less than the second depth from a predetermined reference plane, an internal structure image in which color parameters are continuously changed from the first depth to the second depth is configured, and an internal structure image output unit that causes the internal structure image to be displayed on a display device; A profile image output unit that causes the display device to display a profile image showing the correspondence between the depth from the reference plane and the color parameter; An input reception unit that receives an operation of a pointing device, and is provided with an image processing device. When the operation position of the pointing device is on the profile image, the input reception unit is configured to receive a change in at least one of the first depth and the second depth according to the operation content of the pointing device. The internal structure image output unit is configured to reconstruct the internal structure image in which the color parameter within the depth range after the change received by the input reception unit is changed from the changed first depth to the second depth in the same manner as the change mode of the color parameter before the change, and to cause the internal structure image to be redisplayed on the display device. The profile image output unit is configured to reconstruct the profile image according to the changed first depth and the second depth received by the input reception unit, and to cause the profile image to be redisplayed on the display device.

Effects of the Invention

[0008] According to the present invention, a graphical user interface (GUI) can be provided that makes it easier to recognize the internal structure of an object including a living body.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0010] <The First Embodiment of the Present Invention> Hereinafter, the first embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to these examples, and is shown by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0011] (1) Configuration of the Image Processing Apparatus 10 The image processing apparatus 10 according to the present embodiment is configured to process volume data showing a three-dimensional internal structure of a living tissue, for example, so that a user can easily recognize the internal structure of the living tissue (subject). FIG. 1 is a schematic configuration diagram of the image processing apparatus 10 according to the present embodiment. As shown in FIG. 1, the image processing apparatus 10 includes, for example, an internal structure image output unit 11, a profile image output unit 12, and an input reception unit 13. As shown in FIG. 1, the internal structure image output unit 11 and the profile image output unit 12 may be, for example, a part of a computer having a data storage unit (SSD / HDD) 101, a general-purpose arithmetic unit (CPU) 102, an image arithmetic unit (GPU) 103, an image processing storage unit (VRAM) 104, a general-purpose processing storage unit (RAM) 105, and a communication unit 106. In the present embodiment, the volume data is obtained by reconstructing the signal data of the photoacoustic wave generated by light irradiation on the living body into an image. The user means a person who operates the image processing apparatus 10.

[0012] The SSD / HDD 101 is configured to store data such as programs related to photoacoustic wave measurement. The CPU 102 is configured to control each part of the image processing apparatus 10 by executing a predetermined program stored in the SSD / HDD 101. The GPU 103 is configured to execute a program related to image processing in cooperation with the CPU 102. The VRAM 104 is configured to temporarily hold information, programs, etc. processed by the GPU 103. The RAM 105 has, for example, a volume data memory (area) 107, an internal structure image memory (area) 108, a profile image memory (area) 109, etc., and is configured to temporarily hold information, programs, etc. processed by the CPU 102. The communication unit 106 is configured to obtain information necessary for image processing from the network file server 110. The network file server 110 is configured to store, for example, imaging data captured by the photoacoustic wave imaging device 111.

[0013] The internal structure image output unit 11 constructs an internal structure image 30 in which color parameters are continuously changed from the first depth to the second depth within a depth range of not less than the first depth and not more than the second depth from a predetermined reference plane based on volume data indicating the three-dimensional internal structure of an object (a living tissue in this embodiment), and is configured to display it on the display device 20. Further, the internal structure image output unit 11 makes the color parameters of the internal structure image 30 constructed based on the volume data within the range less than the first depth the same as the color parameters at the first depth, and makes the color parameters of the internal structure image 30 constructed based on the volume data within the range exceeding the second depth the same as the color parameters at the second depth, thereby constructing the internal structure image 30 and configuring it to be displayed on the display device 20.

[0014] The predetermined reference plane means a plane at a predetermined reference height or a continuous plane that defines the interface of the subject. The internal structure image output unit 11 is preferably configured to switch the predetermined reference plane to a plane at a predetermined reference height or a continuous plane that defines the interface of the subject, reconstruct the internal structure image 30, and be able to redisplay it on the display device 20. When obtaining the continuous plane that defines the interface of the subject, for example, a known method such as the cross simulation described in Patent Document 1 can be used.

[0015] The first depth means, for example, the lower limit (the shallowest position) of the depth range, and the second depth means, for example, the upper limit (the deepest position) of the depth range.

[0016] Volume data is three-dimensional data as an aggregate of voxels. A voxel means a unit region that has physical characteristic information such as density, mass, and oxygen saturation of a unit space region of the subject as predetermined physical parameters. That is, each voxel is composed of a coordinate value and a physical parameter, and each voxel in the volume data corresponds to a unit space region of the subject. Also, the internal structure image 30 means a two-dimensional image composed of a set of pixels for display.

[0017] The color parameter is a parameter that includes at least one of the attributes of hue, chroma, and lightness. The color parameter of each pixel in the internal structure image 30 continuously changes, for example, from red (shallow) to blue (deep) according to the depth from the reference plane of each voxel corresponding to each pixel. The internal structure image output unit 11 is preferably configured to be able to switch which of the attributes (hue, chroma, and lightness) included in the hue parameter is continuously changed according to the depth from the reference plane.

[0018] The internal structure image 30 is an image for two-dimensional display configured based on volume data showing the internal structure of biological tissue. In the present embodiment, the case where the internal structure image 30 is a photoacoustic image configured based on signal data of photoacoustic waves will be described.

[0019] The display device 20 is connected to, for example, the image processing device 10 and is configured to display an internal structure image 30 or the like by executing a predetermined program. The display device 20 is a two-dimensional display device and may be a stereoscopic display device. Specifically, the display device 20 is a liquid crystal display, an organic EL (OLED) display, a head-mounted display, a direct-view stereoscopic display, or the like. Note that the display device 20 may be a part of the image processing device 10.

[0020] The profile image output unit 12 is configured to cause the display device 20 to display a profile image 40 indicating the correspondence between the depth from the reference plane and the color parameters, together with the internal structure image 30.

[0021] Figure 2 is a diagram showing an example of an internal structure image 30 and a profile image 40 displayed on the display device 20 according to the present embodiment. As shown in Figure 2, in the present embodiment, the profile image 40 has a color bar 41 and an index 42. The color bar 41 is, for example, a vertically long bar-shaped image, and the color parameters change from the upper end to the lower end in the same manner as the internal structure image 30. In the color bar 41, for example, the lower end corresponds to the first depth, and the upper end corresponds to the second depth. The index 42 is, for example, a plurality of numerical groups adjacent to the color bar 41, and shows the correspondence between the color parameters of the color bar 41 and the depth from the reference plane. For example, among the numerical groups of the index 42, the lowest numerical value indicates the first depth, and the highest numerical value indicates the second depth. By checking the internal structure image 30 and the profile image 40 displayed on the display device 20, the user can recognize how deep the region (hereinafter also referred to as the region of interest) that the user is interested in is located from the reference plane in the internal structure image 30. Note that in the image processing apparatus 10 of the present embodiment, there are two ways to set the reference plane with a depth of zero. One is a setting method in which the skin surface position of the subject is set to zero. The skin surface position can be obtained, for example, by the cross-simulation process described in Patent Document 1. The other is also a setting method in which, as shown in Patent Document 1, when a plurality of sensors are arranged in a bowl shape so as to surround the subject, the curvature center position of the sensors is set to z = zero with respect to the central axis (z-axis) of the bowl. When this reference is used, the height of the shallowest detection surface is approximately -(minus) 10 mm, and in the photoacoustic wave sensor of the present embodiment, the range from -10 mm to +10 mm has the highest resolution.

[0022] The input reception unit 13 is configured to receive the operation of the pointing device 14. In this specification, the pointing device 14 means, for example, an input device for operating a pointer 50 displayed on the display device 20, and specifically, a mouse, a trackball, a touch panel (a touch sensor that detects position designation, dragging, etc. by the user's touch input), and the like.

[0023] (2) The graphical interface provided by the image processing apparatus 10 The image processing apparatus 10 according to the present embodiment includes a graphical user interface (GUI) that makes it easier to recognize the internal structure of a biological tissue. For example, the user operates the pointer 50 displayed on the display device 20 using the pointing device 14, and the image processing apparatus 10 is configured to perform various processes according to the operation position and operation content of the pointing device 14. Hereinafter, a part of the functions of the GUI provided by the image processing apparatus 10 according to the present embodiment will be described. In this specification, the operation content of the pointing device 14 means various operations such as click, drag, and wheel rotation, and the operation position of the pointing device 14 means the position of the pointer 50 when the above-described operation is started.

[0024] (2-1) Image movement function According to the GUI provided by the image processing apparatus 10 according to the present embodiment, the internal structure image 30 and the profile image 40 displayed on the display device 20 can be moved to arbitrary positions on the display device 20, respectively.

[0025] When the operation position of the pointing device 14 received by the input reception unit 13 is on the internal structure image 30, the internal structure image output unit 11 is configured to display the internal structure image 30 in a movable manner according to the operation content of the pointing device 14 received by the input reception unit 13. Specifically, for example, the internal structure image output unit 11 may be configured to move the internal structure image 30 in the direction in which the pointer 50 moves when a drag operation is performed on the internal structure image 30.

[0026] When the operation position of the pointing device 14 received by the input reception unit 13 is on the profile image 40, the profile image output unit 12 is configured to display the profile image 40 in a movable manner according to the operation content of the pointing device 14 received by the input reception unit 13. Specifically, for example, the profile image output unit 12 may be configured to move the profile image 40 in the direction in which the pointer 50 moves when a drag operation is performed on the profile image 40. Since the user can move the profile image 40 to an arbitrary position as needed, for example, it becomes easier to recognize how deep the region of interest is located from the reference plane. In the present embodiment, the operation position of the pointing device 14 being on the profile image 40 means, for example, starting the operation of the pointing device 14 with the pointer 50 on the color bar 41 (or on the index 42).

[0027] (2-2) Image enlargement / reduction function According to the GUI provided in the image processing apparatus 10 according to the present embodiment, it is possible to enlarge or reduce the internal structure image 30 displayed on the display device 20.

[0028] When the operation position of the pointing device 14 received by the input reception unit 13 is on the internal structure image 30, the internal structure image output unit 11 is configured to display the internal structure image 30 in a manner that enables at least one of enlargement or reduction according to the operation content of the pointing device 14 received by the input reception unit 13. Specifically, for example, the internal structure image output unit 11 may be configured to enlarge the internal structure image 30 when a wheel operation is performed in one direction (for example, the upward direction) on the internal structure image 30, and to reduce the internal structure image 30 when a wheel operation is performed in the other direction (for example, the downward direction).

[0029] (2-3) Depth range change function According to the GUI included in the image processing apparatus 10 according to the present embodiment, it is possible to change at least one of the first depth and the second depth and redisplay the internal structure image 30 and the profile image 40 displayed on the display device 20, respectively.

[0030] When the operation position of the pointing device 14 is on the profile image 40, the input reception unit 13 is configured to receive a change in at least one of the first depth and the second depth according to the operation content of the pointing device 14.

[0031] The internal structure image output unit 11 reconstructs an internal structure image 30 in which the color parameters of each pixel within the changed depth range received by the input reception unit 13 are changed in the same manner as the change pattern of the color parameters before the change from the first depth to the second depth, and is configured to redisplay it on the display device 20. In this specification, the "change pattern" of the color parameter indicates which of the attributes included in the hue parameter is changing continuously. Also, changing the color parameter in the same manner as the change pattern of the color parameter before the change means that, for example, if the color parameter before the change changed from red to blue from the first depth to the second depth, the color parameter after the change also remains unchanged in that it changes from red to blue.

[0032] The profile image output unit 12 is configured to reconstruct the profile image 40 according to the changed first depth and second depth received by the input reception unit 13 and redisplay it on the display device 20. As described above, since the change pattern of the color parameter does not change before and after the change of the depth range, when reconstructing the profile image 40, the image of the color bar 41 may remain as it is, and only the index 42 needs to be changed. The user can more visually recognize and adjust the depth direction information of the region of interest by a simple operation of operating the pointing device 14.

[0033] (2-4) Function for changing the first depth (or the second depth) The depth range change function described in (2-3) can be further finely divided. For example, according to the GUI provided in the image processing apparatus 10 according to the present embodiment, it is possible to change only the first depth (or the second depth) and redisplay the internal structure image 30 and the profile image 40 displayed on the display device 20, respectively.

[0034] When the operation position of the pointing device 14 is on the first area of the profile image 40, the input receiving unit 13 is configured to receive a change in either the first depth or the second depth according to the operation content of the pointing device 14. Here, the first area of the profile image 40 means, for example, the lower part of the color bar 41 (for example, the length from the lower end to 1 / 4). Specifically, the input receiving unit 13 may be configured to receive a change that only increases the first depth when a wheel operation is performed in one direction (for example, the upward direction) at the lower part of the color bar 41, and only decreases the first depth when a wheel operation is performed in the other direction (for example, the downward direction).

[0035] When the operation position of the pointing device 14 is on the second area of the profile image 40, the input receiving unit 13 is configured to receive a change in the other one different from the above-mentioned one of the first depth and the second depth according to the operation content of the pointing device 14. Here, the second area of the profile image 40 means, for example, the upper part of the color bar 41 (for example, the length from the upper end to 1 / 4). Specifically, the input receiving unit 13 may be configured to receive a change that only increases the second depth when a wheel operation is performed in one direction (for example, the upward direction) at the upper part of the color bar 41, and only decreases the second depth when a wheel operation is performed in the other direction (for example, the downward direction).

[0036] (2-5) Parallel movement function of depth range According to the GUI included in the image processing apparatus 10 according to the present embodiment, it is possible to change the first depth and the second depth so as to translate the depth range, and to redisplay the internal structure image 30 and the profile image 40 displayed on the display device 20, respectively.

[0037] When the operation position of the pointing device 14 is on the third region of the profile image 40, the input reception unit 13 is configured to receive a change for moving the depth range at least in one of the deeper direction or the shallower direction without changing the size of the depth range according to the operation content of the pointing device 14. Here, the third region of the profile image 40 means, for example, the central portion of the color bar 41 (for example, the length of 1 / 4 from the center in the vertical direction). Specifically, for example, when a wheel operation is performed in one direction (for example, the upward direction) at the central portion of the color bar 41, the input reception unit 13 may be configured to receive a change for increasing the first depth and the second depth by the same value, and when a wheel operation is performed in the other direction (for example, the downward direction), the input reception unit 13 may be configured to receive a change for decreasing the first depth and the second depth by the same value.

[0038] (2-6) Depth range expansion and contraction function According to the GUI included in the image processing apparatus 10 according to the present embodiment, it is possible to change the first depth and the second depth so as to expand or contract the depth range, and to redisplay the internal structure image 30 and the profile image 40 displayed on the display device 20, respectively. When there is no change in the respective color parameters at the first depth and the second depth, when the depth range is expanded, the change in the color parameter per unit depth decreases, and inspection over a wide depth range becomes possible. On the other hand, similarly when there is no change in the respective color parameters, when the depth range is contracted, the change in the color parameter per unit depth increases, and the difference in depth can be read in detail in a narrow depth range.

[0039] When the operation position of the pointing device 14 is on the profile image 40, the input reception unit 13 is configured to receive a change that enables at least one of enlargement or reduction of the depth range without changing the center of the depth range according to the operation content of the pointing device 14. Specifically, for example, when a right-click operation is performed on the profile image 40 and a wheel operation is performed in one direction (for example, the upward direction), the input reception unit 13 increases the first depth and decreases the second depth by the same value as the increase in the first depth to reduce the depth range. When the wheel operation is performed in the other direction (for example, the downward direction), the input reception unit 13 decreases the first depth and increases the second depth by the same value as the decrease in the first depth to expand the depth range. That is, it may be configured to receive such a change.

[0040] (3) Effects Obtained by this Embodiment According to this embodiment, one or more of the following effects can be obtained.

[0041] (a) According to the GUI included in the image processing apparatus 10 according to this embodiment, the internal structure image 30 and the profile image 40 displayed on the display device 20 can be moved to arbitrary positions on the display device 20, respectively. With this function, the user can move the profile image 40 to an arbitrary position as needed, so that, for example, it becomes easier to recognize how deep the region of interest is located from the reference plane.

[0042] (b) According to the GUI included in the image processing apparatus 10 according to this embodiment, the internal structure image 30 displayed on the display device 20 can be enlarged or reduced. With this function, the user can enlarge or reduce the internal structure image 30 as needed, so that, for example, it becomes easier to recognize how deep the region of interest is located from the reference plane.

[0043] (c) According to the GUI provided in the image processing apparatus 10 according to this embodiment, it is possible to change at least one of the first depth and the second depth and redisplay the internal structure image 30 and the profile image 40 displayed on the display device 20, respectively. With this function, the user can more visually recognize and adjust the depth direction information of the region of interest by a simple operation of operating the pointing device 14.

[0044] (d) According to the GUI provided in the image processing apparatus 10 according to this embodiment, it is possible to change only the first depth (or the second depth) and redisplay the internal structure image 30 and the profile image 40 displayed on the display device 20, respectively. With this function, the user can more visually recognize and adjust the depth direction information of the region of interest by a simple operation of operating the pointing device 14.

[0045] (e) According to the GUI provided in the image processing apparatus 10 according to this embodiment, it is possible to change the first depth and the second depth so as to translate the depth range and redisplay the internal structure image 30 and the profile image 40 displayed on the display device 20, respectively. With this function, the user can more visually recognize and adjust the depth direction information of the region of interest by a simple operation of operating the pointing device 14.

[0046] (f) According to the GUI provided in the image processing apparatus 10 according to this embodiment, it is possible to change the first depth and the second depth so as to expand or contract the depth range and redisplay the internal structure image 30 and the profile image 40 displayed on the display device 20, respectively. With this function, the user can more visually recognize and adjust the depth direction information of the region of interest by a simple operation of operating the pointing device 14.

[0047] (g) In this embodiment, the volume data showing the internal structure of the biological tissue is obtained by reconstructing the signal data of the photoacoustic wave generated by irradiating light on the biological body. In photoacoustic tomography, for example, since it is possible to obtain an internal structure image 30 that visualizes only blood vessels and simultaneously displays the blood vessels in the foreground and the blood vessels in the background, the present invention can be particularly effectively applied.

[0048] (4-1) Modification Example 1 of the First Embodiment The above-described embodiment can be modified as shown in the following modification examples as necessary. Hereinafter, only the elements different from the above-described embodiment will be described, and the same reference numerals will be given to the elements substantially the same as those described in the above-described embodiment, and the description thereof will be omitted.

[0049] FIG. 3 is a schematic configuration diagram of the image processing apparatus 10 according to this modification example. As shown in FIG. 3, the image processing apparatus 10 according to this modification example may further include, for example, a slider image output unit 15. The slider image output unit 15 may be a part of a computer having a CPU 102 or the like, similar to the internal structure image output unit 11 and the profile image output unit 12. The slider image output unit 15, for example, for the entire depth range (also referred to as the imaging range) of the three-dimensional volume data obtained by imaging, the position and size of the depth range (from the first depth to the second depth) in which the color parameters continuously change, a first slider image 60, and for the entire depth range of the volume data, the depth position and the size of the depth range of the display range (in the depth direction) of the internal structure image 30 displayed on the display device 20, a second slider image 70, are configured to be displayed on the display device 20 together with the internal structure image 30 and the profile image 40.

[0050] FIG. 4 is a diagram showing an example of an internal structure image 30, a profile image 40, a first slider image 60, and a second slider image 70 displayed on the display device 20 according to this modified example. As shown in FIG. 4, the first slider image 60 is, for example, a horizontally long bar-shaped image and has a first slider bar 61 therein. Also, the second slider image 70 is, for example, a horizontally long bar-shaped image and has a second slider bar 71 therein. In the first slider image 60 and the second slider image 70, one end (for example, the left end) indicates the lower limit (the shallowest position) of the depth range of the volume data, and the other end (for example, the right end) indicates the upper limit (the deepest position) of the depth range of the volume data. In the first slider bar 61, one end (for example, the left end) corresponds to the first depth, and the other end (for example, the right end) corresponds to the second depth. Also, in the second slider bar 71, one end (for example, the left end) corresponds to the lower limit of the display range of the internal structure image 30, and the other end (for example, the right end) corresponds to the upper limit of the display range of the internal structure image 30.

[0051] The slider image output unit 15 is configured to reconstruct the first slider image 60 according to the changed first depth and second depth received by the input reception unit 13 and redisplay it on the display device 20. Specifically, the slider image output unit 15 may be configured to change the position and size of the first slider bar 61 according to the changed first depth and second depth. By checking the first slider image 60 as needed, the user can easily recognize where in the entire depth range of the volume data the depth range in which the color parameters are continuously changing is located.

[0052] The first slider image 60 can be used as part of a GUI having a depth range change function, similar to the profile image 40. In this case, the input reception unit 13 may be configured to receive a change in at least one of the first depth and the second depth according to the operation content of the pointing device 14 when the operation position of the pointing device 14 is on the first slider image 60. However, from the viewpoint that the change in the depth range can be performed while visually confirming the correspondence between the depth from the reference plane and the color parameters, it is preferable to operate the pointing device 14 on the profile image 40 as in the first embodiment described above.

[0053] The second slider image 70 can be used as part of a GUI having a function of changing the display range of the internal structure image 30. In this case, the input reception unit 13 may be configured to receive a change in at least one of the lower limit and the upper limit of the display range of the internal structure image 30 according to the operation content of the pointing device 14 when the operation position of the pointing device 14 is on the second slider image 70. Also, the internal structure image output unit 11 may be configured to reconstruct the internal structure image 30 according to the changed display range of the internal structure image 30 received by the input reception unit 13 and re-display it on the display device 20. Further, the slider image output unit 15 may be configured to reconstruct the second slider image 70 according to the changed display range of the internal structure image 30 received by the input reception unit 13 and re-display it on the display device 20. Specifically, the slider image output unit 15 may be configured to change the position and size of the second slider bar 71 according to the changed display range of the internal structure image 30. By changing the display range of the internal structure image 30 as needed, the user can, for example, make the voxels in the unnecessary area displayed in the foreground invisible and make it easier to recognize the information in the region of interest.

[0054] (4-2) Modification Example 2 of the First Embodiment FIG. 7 is a schematic configuration diagram of the image processing apparatus 10 according to this modification example. As shown in FIG. 7, the image processing apparatus 10 of this modification example includes, for example, an image processor 100, a display device 20, an input reception unit 13, and a pointing device 14. The image processor 100 includes, for example, an internal structure image output unit 11, a profile image output unit 12, a data storage unit (SSD / HDD) 101, and a communication unit 106. The internal structure image output unit 11 is composed of, for example, an image calculation unit (GPU) 103, an image processing storage unit (VRAM) 104, a volume data area 107 and an internal structure image area 108 in the VRAM 104. The profile image output unit 12 is composed of, for example, a general-purpose calculation unit (CPU) 102, a general-purpose processing storage unit (RAM) 105, the VRAM 104, and a profile image area 109 in the VRAM 104. The volume data obtained by imaging with the photoacoustic imaging device 111 is stored in the network file server 110. When the user wants to observe the volume data, the volume data is stored in the volume data area 107 in the VRAM 104 via the communication unit 106, the data storage unit 101, and the CPU 102. The process of converting the volume data into an internal structure image and the user's instruction via the input reception unit 13 are sent to the GPU 103 via the CPU 102, and the internal structure image is processed according to the instruction and stored in the internal structure image area 108. The information of the profile image 40 is directly written by the CPU 102 into the profile image area 109 in the VRAM 104. The contents of the internal structure image area 108 and the profile image area 109 are displayed on the display device 20.

[0055] The image processor 100 includes the components described below. The data storage unit 101 is configured to store data such as a program related to photoacoustic wave measurement. The CPU 102 is configured to control each part of the image processing apparatus 10 by executing a predetermined program stored in the data storage unit 101. The GPU 103 is configured to execute a program related to image processing in cooperation with the CPU 102. The VRAM 104 is a memory for displaying the image information processed by the GPU 103 and the CPU 102 on the display device 20. The VRAM 104 is also used as a working memory for image processing in the GPU 103. The RAM 105 is a working memory for the CPU 102 and is configured to temporarily hold information, programs, etc. processed by the CPU 102. The communication unit 106 is configured to acquire information necessary for image processing from the network file server 110 and store it in the data storage unit 101. The network file server 110 is configured to store, for example, the imaging data captured by the photoacoustic imaging device 111.

[0056] (5) Operation examples of the graphical interface Next, operation examples of the graphical interface according to the present invention will be described. The following operation examples are an example of the present invention, and the present invention is not limited by these operation examples.

[0057] FIG. 5A is a diagram showing an example of an internal structure image 30 and a profile image 40 before changing the depth range according to this operation example. In this operation example, the internal structure image 30 is a photoacoustic image showing the internal blood vessels of the subject's foot. The reference plane is an example of a continuous plane defined with the vicinity of the skin surface of the subject as the reference position by the cross simulation described above.

[0058] As shown in FIG. 5A, before the depth range was changed, the first depth was 0 mm and the second depth was 4 mm. By corresponding and checking the color parameters of the internal structure image 30 and the profile image 40, the user can recognize, for example, how deep the blood vessels existing in the region of interest within the depth range of not less than the first depth and not more than the second depth are located from the reference plane.

[0059] For example, when the region of interest is in a deeper range (for example, about 5 mm), the user may operate the pointing device 14, move the pointer 50 to the center of the color bar 41, and perform a wheel operation upward. By the above operation, the first depth and the second depth can be increased by the same value, and the depth range can be changed (translated parallel). The internal structure image 30 and the profile image 40 after the depth range is changed by the above operation are shown in FIG. 5B. As shown in FIG. 5B, after the depth range was changed, the first depth was 2 mm and the second depth was 6 mm. By corresponding and checking the color parameters of the redisplayed internal structure image 30 and the profile image 40, the user can recognize, for example, how deep the blood vessels existing in the region of interest in a deeper range are located from the reference plane.

[0060] However, in the example shown in FIG. 5B, since the depth range has been translated, for example, for blood vessels existing in the depth range from 0 mm to 2 mm or less from the reference plane, the change in the color parameter in the internal structure image 30 showing the depth of each voxel constituting the blood vessel region is lost and becomes constant (the same value as the first depth after the change). Therefore, it becomes difficult for the user to recognize how deep the blood vessels existing in this range are located from the reference plane. For example, when the region of interest is a deeper range while maintaining the change in the color parameter in the shallower range, the user may operate the pointing device 14, move the pointer 50 to the upper part of the color bar 41, and perform a wheel operation upward. By the above operation, only the second depth can be increased and the depth range can be changed. FIG. 5C shows the internal structure image 30 and the profile image 40 after the depth range is changed by the above operation. As shown in FIG. 5C, after the depth range is changed, the first depth is 0 mm and the second depth is 6 mm. By corresponding and checking the color parameter of the redisplayed internal structure image 30 and the profile image 40, the user can recognize, for example, how deep the blood vessels existing in the region of interest in a deeper range and the blood vessels existing in the depth range before the change are located from the reference plane.

[0061] Also, for example, when the region of interest is a narrower depth range, the user may operate the pointing device 14, move the pointer 50 onto the color bar, perform a right-click operation, and perform a wheel operation upward. By the above operation, the first depth can be increased, and the second depth can be decreased by the same value as the increase in the first depth, and the depth range can be changed (narrowed). FIG. 5D shows the internal structure image 30 and the profile image 40 after the depth range is changed by the above operation. As shown in FIG. 5D, after the depth range is changed, the first depth is 0.8 mm and the second depth is 2.4 mm. By corresponding and checking the color parameter of the redisplayed internal structure image 30 and the profile image 40, the user can recognize in more detail, for example, how deep the blood vessels existing in the region of interest in a narrower depth range are located from the reference plane.

[0062] As described above, it was confirmed that the user can more visually and easily recognize and adjust the depth direction information of the biological tissue by the GUI included in the image processing apparatus 10.

[0063] (6) Image processing method by the image processing apparatus 10 Next, the image processing method by the image processing apparatus 10 will be described. FIG. 6 is a flowchart showing an outline of the image processing method by the image processing apparatus 10 according to the first embodiment of the present invention. As shown in FIG. 6, the image processing method of the present embodiment includes, for example, a volume data acquisition step S100, a depth information and color parameter acquisition step S110, an internal structure image display step S120, a profile image display step S130, an input determination step S140, an operation position determination step S150, and a processing step S160.

[0064] In the volume data acquisition step S100, for example, the photoacoustic imaging device 111 is activated to perform photoacoustic imaging, and volume data indicating the three-dimensional internal structure of an object (for example, a biological tissue) is obtained.

[0065] In the depth information and color parameter acquisition step S110, for example, information on the first depth and the second depth from a predetermined reference plane, and color parameters of the internal structure image 30 constructed based on the volume data within the depth range from the first depth to the second depth are obtained.

[0066] In the internal structure image display step S120, for example, based on the volume data and the color parameters, an internal structure image 30 in which the color parameters continuously change from the first depth to the second depth is configured and displayed on the display device 20.

[0067] In the profile image display step S130, for example, a profile image 40 showing the correspondence between the depth from the reference plane and the color parameters is displayed on the display device 20.

[0068] In the input determination step S140, for example, it is determined whether there is an input from the pointing device 14. If there is an input from the pointing device 14, the operation position determination step S150 is performed. If there is no input from the pointing device 14, the input determination step S140 may be performed again.

[0069] In the operation position determination step S150, for example, it is determined whether the operation position of the pointing device 14 is on the profile image 40. If the operation position of the pointing device 14 is on the profile image 40, for example, the depth information and color parameter acquisition step S110 may be performed again. If the operation position of the pointing device 14 is not on the profile image 40, the processing step S160 may be performed.

[0070] In the processing step S160, other processing is performed according to the operation content of the pointing device 14.

[0071] Through the above steps, image processing can be performed to make it easier to recognize the internal structure of the living tissue. Note that the present invention is also applicable as a program for causing a computer to execute the above steps (procedures) and a computer-readable recording medium on which the program is recorded.

[0072] <Other Embodiments of the Present Invention> The embodiments of the present invention have been specifically described above. However, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist thereof.

[0073] For example, in the above-described embodiment, the volume data indicating the internal structure of the biological tissue includes the signal data of the photoacoustic wave generated by irradiating light on the biological body, and the internal structure image 30 is described as a photoacoustic image configured based on the signal data of the photoacoustic wave. However, the internal structure image 30 is not limited to the photoacoustic image. The internal structure image 30 may be, for example, an ultrasonic image configured based on data obtained by ultrasonic diagnostic imaging, or an MRI angiogram image configured based on data obtained by MRI angiography diagnostic method. Further, in the above-described embodiment, the case of processing the volume data indicating the three-dimensional internal structure of the biological tissue is described. However, the present invention is applicable not only to biological bodies but also to analysis devices that analyze the internal structure of roads with ultrasonic waves and the like.

[0074] <Preferred Embodiment of the Present Invention> Hereinafter, preferred embodiments of the present invention will be appended.

[0075] (Appendix 1) Based on volume data indicating the three-dimensional internal structure of an object, within a depth range from a first depth to a second depth that is equal to or greater than the first depth from a predetermined reference plane, an internal structure image in which color parameters are continuously changed from the first depth to the second depth is configured and displayed on a display device, an internal structure image output unit for A profile image output unit that displays on the display device a profile image showing the correspondence between the depth from the reference plane and the color parameter; An input reception unit that receives an operation of a pointing device, and When the operation position of the pointing device is on the profile image, the input reception unit is configured to receive a change in at least one of the first depth and the second depth according to the operation content of the pointing device. The internal structure image output unit is configured to reconstruct the internal structure image in which the color parameters within the changed depth range received by the input reception unit are changed from the changed first depth to the second depth in the same manner as the change pattern of the color parameters before the change, and to re-display the reconstructed internal structure image on the display device. The profile image output unit is configured to reconstruct the profile image according to the changed first depth and the second depth received by the input reception unit, and to re-display the reconstructed profile image on the display device. An image processing apparatus.

[0076] (Appendix 2) When the operation position of the pointing device received by the input reception unit is on the internal structure image, the internal structure image output unit is configured to display the internal structure image in a movable manner according to the operation content of the pointing device received by the input reception unit. When the operation position is on the profile image, the profile image output unit is configured to display the profile image in a movable manner according to the operation content. The image processing apparatus according to Appendix 1.

[0077] (Appendix 3) When the operation position of the pointing device received by the input reception unit is on the internal structure image, the internal structure image output unit is configured to display the internal structure image in a manner that allows at least one of enlargement or reduction according to the operation content of the pointing device received by the input reception unit. The image processing apparatus according to Appendix 1 or Appendix 2.

[0078] (Appendix 4) When the operation position of the pointing device is on the first area of the profile image, the input reception unit is configured to receive a change in either the first depth or the second depth according to the operation content of the pointing device. The image processing apparatus according to any one of Appendices 1 to 3.

[0079] (Supplementary Note 5) When the operation position of the pointing device is on the second region of the profile image, the input receiving unit is configured to receive a change of the other one different from the one of the first depth and the second depth according to the operation content of the pointing device. The image processing apparatus according to Supplementary Note 4.

[0080] (Supplementary Note 6) When the operation position of the pointing device is on the third region of the profile image, the input receiving unit is configured to receive a change of moving the depth range in at least one of the deeper direction or the shallower direction without changing the size of the depth range according to the operation content of the pointing device. The image processing apparatus according to any one of Supplementary Notes 1 to 5.

[0081] (Supplementary Note 7) When the operation position of the pointing device is on the profile image, the input receiving unit is configured to receive a change enabling at least one of expansion or contraction of the depth range without changing the center of the depth range according to the operation content of the pointing device. The image processing apparatus according to any one of Supplementary Notes 1 to 6.

[0082] (Supplementary Note 8) The volume data is obtained by reconstructing an image of a signal data of a photoacoustic wave generated by light irradiation on a living body. The image processing apparatus according to any one of Supplementary Notes 1 to 7.

[0083] (Supplementary Note 9) A step of obtaining volume data indicating a three-dimensional internal structure of an object; A step of obtaining information on a first depth and a second depth from a predetermined reference plane, and color parameters of an internal structure image based on the volume data within the depth range from the first depth to the second depth; Based on the volume data and the color parameters, a step of forming the internal structure image in which the color parameters are continuously changed from the first depth to the second depth and displaying the image on a display device; A step of displaying a profile image showing the correspondence between the depth from the reference plane and the color parameters on the display device; A step of determining an input of a pointing device; A step of determining an operation position of the pointing device; An image processing method comprising:

[0084] (Appendix 10) A procedure for obtaining volume data indicating a three-dimensional internal structure of an object; A procedure for obtaining information on a first depth and a second depth from a predetermined reference plane and color parameters of an internal structure image based on the volume data within a depth range from the first depth to the second depth; Based on the volume data and the color parameters, a step procedure of forming the internal structure image in which the color parameters are continuously changed from the first depth to the second depth and displaying the image on a display device; A procedure of displaying a profile image showing the correspondence between the depth from the reference plane and the color parameters on the display device; A procedure of determining an input of a pointing device; A procedure of determining an operation position of the pointing device; A computer-readable recording medium recording a program for causing a computer to execute the following:

[0085] (Appendix 11) A step of causing an internal structure image output unit to form an internal structure image in which color parameters are continuously changed from the first depth to the second depth within a depth range of not less than the first depth and not more than the second depth from a predetermined reference plane based on volume data indicating a three-dimensional internal structure of an object, and displaying the image on a display device; A step of causing a display device to display a profile image showing a correspondence relationship between a depth from the reference plane and the color parameter by a profile image output unit; A step of causing an input reception unit to receive an operation of a pointing device; When the operation position of the pointing device is on the profile image, a step of causing the input reception unit to receive a change in at least one of the first depth and the second depth according to the operation content of the pointing device; A step of causing the internal structure image output unit to reconstruct the internal structure image in which the color parameter within the depth range after the change received by the input reception unit is changed from the changed first depth to the second depth in the same manner as the change mode of the color parameter before the change, and causing the display device to redisplay it; A step of causing the profile image output unit to reconstruct the profile image according to the changed first depth and second depth received by the input reception unit and causing the display device to redisplay it; An image processing method further having the above.

[0086] (Appendix 12) On a computer, Based on volume data showing a three-dimensional internal structure of an object, an internal structure image output unit that constructs an internal structure image in which a color parameter is continuously changed from the first depth to the second depth within a depth range from the first depth to the second depth with respect to a predetermined reference plane and causes the display device to display it; A profile image output unit that causes the display device to display a profile image showing a correspondence relationship between a depth from the reference plane and the color parameter; An input reception unit that receives an operation of a pointing device is realized, The input reception unit is configured to receive a change in at least one of the first depth and the second depth according to the operation content of the pointing device when the operation position of the pointing device is on the profile image; Configure the internal structure image output unit to reconstruct the internal structure image in which the color parameters within the changed depth range received by the input reception unit are changed from the changed first depth to the second depth in the same manner as the change pattern of the color parameters before the change, and re-display the reconstructed image on the display device. Configure the profile image output unit to reconstruct the profile image according to the changed first depth and the second depth received by the input reception unit, and re-display the reconstructed image on the display device. A computer-readable recording medium storing a program.

Explanation of Signs

[0087] 10 Image processing apparatus 11 Internal structure image output unit 12 Profile image output unit 13 Input reception unit 14 Pointing device 15 Slider image output unit 20 Display device 30 Internal structure image 40 Profile image 41 Color bar 42 Index 50 Pointer 60 First slider image 61 First slider bar 70 Second slider image 71 Second slider bar 100 Image processor 101 Data storage unit (SSD / HDD) 102 General-purpose arithmetic unit (CPU) 103 Image arithmetic unit (GPU) 104 Image processing storage unit (VRAM) 105 General-purpose processing storage unit (RAM) 106 Communication unit 107 Volume data memory (area) 108 Internal structure image memory (area) 109 Profile image memory (area) 110 Network file server 111 Photoacoustic imaging device S100 Volume data acquisition step S110 Depth information and color parameter acquisition step S120 Internal structure image display step S130 Profile image display step S140 Input judgment step S150 Operation position judgment step S160 Processing step

Claims

1. Based on volume data showing the three-dimensional internal structure of an object, within a depth range from a first depth to a second depth that is equal to or greater than the first depth from a predetermined reference plane, an internal structure image is configured in which color parameters are continuously changed from the first depth to the second depth, and an internal structure image output unit that causes the image to be displayed on a display device; A profile image output unit that causes the display device to display a profile image having a numerical group indicating the depth and an image in which the color parameters are changed, in order to show the correspondence between the depth from the reference plane and the color parameters; An input reception unit that receives an operation of a pointing device capable of wheel operation; and When the operation position of the pointing device is on a first region in the shallower part of the profile image, the input reception unit accepts a change in the first depth in response to the wheel operation of the pointing device. When the operation position of the pointing device is on a second region in the deeper part of the profile image, the input reception unit accepts a change in the second depth in response to the wheel operation of the pointing device. When the operation position of the pointing device is on a third region in the central part of the profile image, the input reception unit accepts a change in which the first depth and the second depth are changed by the same value in response to the wheel operation of the pointing device, and the depth range is moved in at least one of the deeper direction or the shallower direction without changing the size of the depth range; The internal structure image output unit is configured to reconstruct the internal structure image in which the color parameters within the depth range after the change received by the input reception unit are changed from the changed first depth to the second depth in the same manner as the change pattern of the color parameters before the change, and cause the image to be redisplayed on the display device; The profile image output unit is configured to reconstruct the numerical group of the profile image according to the first depth and the second depth after the change received by the input reception unit, and cause the display device to redisplay the profile image without changing the image in which the color parameters are changed. An image processing apparatus.

2. When the operation position of the pointing device received by the input reception unit is on the internal structure image, the internal structure image output unit is configured to display the internal structure image in a movable manner according to the operation content of the pointing device received by the input reception unit. The profile image output unit is configured to display the profile image in a movable manner according to the operation content when the operation position is on the profile image. The image processing apparatus according to claim 1. **Claim 3** When the operation position of the pointing device received by the input reception unit is on the internal structure image, the internal structure image output unit is configured to display the internal structure image in a manner that enables at least one of enlargement or reduction according to the operation content of the pointing device received by the input reception unit. The image processing apparatus according to claim 1. **Claim 4** The pointing device can perform a click operation. When the operation position of the pointing device is on the profile image in a state where a click operation is performed, the input reception unit is configured to receive a change that enables at least one of enlargement or reduction of the depth range without changing the center of the depth range according to the operation content of the pointing device. The image processing apparatus according to any one of claims 1 to 3. **Claim 5** The volume data is obtained by reconstructing an image of a signal data of a photoacoustic wave generated by light irradiation on a living body. The image processing apparatus according to any one of claims 1 to 4. **Claim 6** The reference plane is a continuous plane that defines an interface of the object. The image processing apparatus according to any one of claims 1 to 5. **Claim 7** A step of configuring, by an internal structure image output unit, an internal structure image in which color parameters are continuously changed from the first depth to the second depth within a depth range from the first depth to the second depth or more based on volume data indicating a three-dimensional internal structure of an object, and displaying the internal structure image on a display device; A step of displaying, by a profile image output unit, a profile image having a numerical group indicating the depth and an image in which color parameters are changed on the display device in order to show a correspondence relationship between the depth from the reference plane and the color parameters. a step of causing an input reception unit to receive an operation of a pointing device capable of wheel operation; when the operation position of the pointing device is on a first region of a shallow portion of the profile image by the input reception unit, in response to a wheel operation of the pointing device, receiving a change in the first depth, and when the operation position of the pointing device is on a second region of a deep portion of the profile image, in response to a wheel operation of the pointing device, receiving a change in the second depth, and when the operation position of the pointing device is on a third region of a central portion of the profile image, in response to a wheel operation of the pointing device, changing the first depth and the second depth by the same value, and receiving a change for moving the depth range in at least one of a deeper direction or a shallower direction without changing the size of the depth range; a step of causing the internal structure image output unit to reconstruct the internal structure image in which color parameters within the changed depth range received by the input reception unit are changed in the same manner as the change pattern of the color parameters before the change from the changed first depth to the second depth, and to re-display the reconstructed internal structure image on the display device; a step of causing the profile image output unit to reconstruct the numerical group of the profile image according to the changed first depth and the second depth received by the input reception unit, and to re-display the reconstructed profile image on the display device without changing the image in which the color parameters are changed; An image processing method further comprising the above steps.

8. On a computer, Based on volume data showing a three-dimensional internal structure of an object, an internal structure image output unit that constructs an internal structure image in which color parameters are continuously changed from a first depth to a second depth within a depth range of not less than the first depth and not more than the second depth from a predetermined reference plane, and displays the constructed internal structure image on a display device; a profile image output unit that displays on the display device a profile image having a numerical group indicating the depth and an image in which color parameters are changed, in order to show a correspondence relationship between the depth from the reference plane and the color parameters; an input reception unit that receives an operation of a pointing device capable of wheel operation; When the operation position of the pointing device is on the first region of the light part of the profile image, the input reception unit receives a change in the first depth in response to a wheel operation of the pointing device; when the operation position of the pointing device is on the second region of the deep part of the profile image, the input reception unit receives a change in the second depth in response to a wheel operation of the pointing device; when the operation position of the pointing device is on the third region of the central part of the profile image, the input reception unit receives a change that changes the first depth and the second depth by the same value in response to a wheel operation of the pointing device, and moves the depth range in at least one of the deeper direction or the shallower direction without changing the size of the depth range, The internal structure image output unit is configured to reconstruct the internal structure image in which the color parameters within the depth range after the change received by the input reception unit are changed in the same manner as the change mode of the color parameters before the change from the changed first depth to the second depth, and to redisplay the reconstructed internal structure image on the display device. The profile image output unit is configured to reconstruct the numerical group of the profile image according to the first depth and the second depth after the change received by the input reception unit, and to redisplay the reconstructed profile image on the display device without changing the image in which the color parameters are changed. A computer-readable recording medium having a program recorded thereon.

Images