Program and information processing method

Abstract

A program according to one aspect of the present disclosure causes a computer to execute: a step for acquiring two or more target image groups for an object; a step for displaying, with respect to a target cross section, corresponding target images included in the two or more target image groups, a template shape, and one or more corresponding control points on a display; a step for receiving a movement operation with respect to the displayed one or more control points in the target cross section; a step for deforming the template shape in accordance with the arrangement of the control points of each control point group after the movement operation; and a step for outputting the deformed template shape as an annotation for the structure of the object after the movement operation is completed.

Description

Program and information processing method

[0001] The present disclosure relates to a program and an information processing method.

[0002] Because muscle mass is an important determinant of force-generating capacity, quantifying individual muscle mass in vivo has attracted significant interest in various fields, including health, medicine, and sports. Muscle mass is assessed by segmenting muscles from magnetic resonance (MR) images, computed tomography (CT) images, and other methods. Traditionally, segmentation has been performed manually by experts. However, tracing the contours of each muscle slice by slice can be extremely time-consuming, especially when there are a large number of muscles. Therefore, in recent years, technologies have emerged that automate the segmentation process using pre-trained image analysis models generated by machine learning. This artificial intelligence-based segmentation method shows great promise.

[0003] Special table 2018-529475 publication

[0004] When adopting an artificial intelligence-based segmentation method, training a new model requires generating a dataset consisting of a combination of training data and ground truth labels. For example, Patent Literature 1 proposes a system for annotating medical images using input from a touch panel. However, this annotation method requires the operator to have specialized knowledge, and correctly tracing the musculoskeletal system is difficult and time-consuming. Note that this problem does not occur only when annotating medical images. It can occur in various situations where annotating the structure of an object is required.

[0005] In one aspect, the present disclosure has been made in view of the above circumstances, and an object thereof is to provide a technology for providing annotations in an easy manner.

[0006] In order to solve the above-mentioned problems, the present disclosure employs the following configurations. Note that the following configurations can be combined as appropriate.

[0007] A program according to one aspect of the present disclosure is a program for causing a computer to execute an information processing method. Two or more cross-section groups are defined for two or more directions, respectively. Two or more control point groups are independently defined corresponding to the two or more cross-section groups. Each cross-section group includes one or more cross sections in the defined direction. Each control point group includes one or more control points arranged for each of the cross sections included in the corresponding cross-section group. The information processing method includes the steps of: acquiring two or more target image groups of an object, each of the two or more target image groups corresponding to a respective one of the two or more defined cross-section groups; displaying, on a display, a corresponding target image included in the acquired two or more target image groups, a template shape indicating a division of the object's structure, and one or more corresponding control points included in a corresponding control point group, for a target cross-section among the one or more cross-sections included in the cross-section group defined for a target direction among the two or more directions; accepting a movement operation for the one or more control points in the target cross-section; deforming the template shape in accordance with the arrangement of the one or more control points included in each control point group after the movement operation; and, after the movement operation is completed, outputting the deformed template shape as an annotation for the object's structure shown in the two or more target image groups.

[0008] In this configuration, a template shape is assigned to each segment of the object's structure. However, because the structure of an object may vary from one object to another, accurate annotation is difficult simply by assigning a template shape. Therefore, cross-section groups are defined in two or more directions, and annotation of the object's structure involves deformation of the template shape by moving each control point on each cross-section in each direction. This deformation can be performed intuitively by moving the control points to deform the template shape so that it matches the object's structure depicted in the target image of the target cross-section. Therefore, this deformation operation is easy. However, if a control point group is assigned in common to all directions, a control point moved on a cross-section in one direction may disappear from the cross-section group in that direction if it moves away from the cross-section in the other direction. If the control point disappears, the task may become difficult (i.e., the benefit of this easy task may not be realized). In contrast, in this configuration, by assigning a control point group independently to each direction, it is possible to prevent the control points from disappearing from the cross-section in each direction. Therefore, this configuration enables annotation to be assigned in an easy manner.

[0009] In the program according to the above aspect, the object may be biological tissue. Each of the target image groups may be one of a magnetic resonance image group, a computed tomography image group, a micro-computed tomography image group, an ultrasound image group, and an optical coherence tomography image group. This configuration allows annotations to be added in a simple manner when each image group is used to observe biological tissue.

[0010] In the program according to the above aspect, the biological tissue may be a musculoskeletal tissue. With this configuration, musculoskeletal annotations can be added in a simple manner.

[0011] In the program according to the above aspect, the two or more cross-sectional groups may include a horizontal plane group, a sagittal plane group, and a coronal plane group. With this configuration, annotations can be easily added when observing an object in the horizontal plane, the sagittal plane, and the coronal plane.

[0012] Note that the form of the present disclosure is not limited to the above program. As another aspect of the program according to each of the above aspects, one aspect of the present disclosure may be an information processing device that realizes all or part of each of the above configurations, an information processing method, or a storage medium that stores the above program and is readable by a machine such as a computer. A storage medium that is readable by a machine such as a computer is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

[0013] For example, an information processing method according to an aspect of the present disclosure may be an information processing method executed by a computer. The information processing method may include the steps of: acquiring two or more target image groups of an object, each of the two or more target image groups corresponding to one of the two or more defined cross-section groups; displaying, on a display, a corresponding target image included in the acquired two or more target image groups, a template shape indicating a section of the object's structure, and one or more corresponding control points included in a corresponding control point group, for the one or more target cross-sections included in the cross-section group defined for the target direction among the two or more directions; accepting a movement operation for the one or more control points in the target cross-section; deforming the template shape according to an arrangement of the one or more control points after the movement operation; and outputting, after the movement operation is completed, the deformed template shape as an annotation for the object's structure shown in the two or more target image groups.

[0014] According to the present disclosure, annotations can be added in an easy manner.

[0015] FIG. 1 schematically illustrates an example of a scenario to which the present disclosure is applied. FIG. 2 schematically illustrates an example of a template, a target image, and a control point set before deformation according to this embodiment. FIG. 3 schematically illustrates an example of a template, a target image, and a control point set after deformation according to this embodiment. FIG. 4A schematically illustrates an example of an initial state of a control point group defined for a group of horizontal planes. FIG. 4B schematically illustrates an example of a target image and a control point set for a horizontal plane in the initial state. FIG. 4C schematically illustrates an example of an arrangement of control points observed on a sagittal plane, which are included in a control point group defined corresponding to a target image on a sagittal plane and a group of horizontal planes in the initial state. FIG. 4D schematically illustrates an example of an arrangement of control points observed on a coronal plane, which are included in a control point group defined corresponding to a target image on a coronal plane and a group of horizontal planes in the initial state. FIG. 5A schematically illustrates an example of a control point movement operation on the horizontal plane of FIG. 4B. FIG. 5B schematically illustrates an example of a state of a control point group defined for a group of horizontal planes after the control point movement operation of FIG. 5A. FIG. 5C schematically illustrates an example of the arrangement of control points included in the control point group defined corresponding to the horizontal plane group, observed on a sagittal plane, after moving the control points of FIG. 5A . FIG. 5D schematically illustrates an example of the arrangement of control points included in the control point group defined corresponding to the horizontal plane group, observed on a coronal plane, after moving the control points of FIG. 5A . FIG. 6A schematically illustrates an example of the control point group defined for the sagittal plane group and the control points on the sagittal plane, after moving the control points of FIG. 5A . FIG. 6B schematically illustrates an example of the control point group defined for the coronal plane group and the control points on the coronal plane, after moving the control points of FIG. 5A . FIG. 7 schematically illustrates an example of the hardware configuration of an information processing device according to this embodiment. FIG. 8 schematically illustrates an example of the software configuration of an information processing device according to this embodiment. FIG. 9 is a flowchart illustrating an example of a processing procedure of the information processing device according to this embodiment.

[0016] An embodiment according to one aspect of the present disclosure (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present disclosure in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present disclosure. In other words, when implementing the present disclosure, specific configurations according to the embodiment may be appropriately adopted. Note that, although data appearing in the present embodiment is described in natural language, more specifically, it is specified in computer-recognizable pseudo-language, commands, parameters, machine language, etc.

[0017] §1 Application Example FIG. 1 schematically illustrates an example of a scenario in which the present disclosure is applied. An information processing device 1 according to this embodiment is one or more computers configured to accept annotation work on the structure of an object. In this embodiment, two or more cross-section groups are defined for two or more directions, respectively. Two or more control point groups are independently defined corresponding to the two or more cross-section groups, respectively. Each cross-section group includes one or more cross sections in the defined direction. Each control point group includes one or more control points arranged for each cross section included in the corresponding cross-section group.

[0018] The information processing device 1 is connected to a display 141. The information processing device 1 acquires two or more target image groups of an object. Each of the two or more target image groups corresponds to one of two or more defined cross-section groups. The information processing device 1 displays, on the display 141, a corresponding target image included in the acquired two or more target image groups, a template shape indicating a division of the object's structure, and one or more corresponding control points included in the corresponding control point group for one or more target cross-sections included in the cross-section group defined for the target direction among two or more directions. In one example, the information processing device 1 selects one direction from two or more directions as the target direction. The information processing device 1 selects one cross-section from the cross-section group defined for the selected target direction as the target cross-section. Then, the information processing device 1 displays, on the display 141, the target image corresponding to the selected target cross-section, a partial shape (i.e., a cross-sectional shape) on the target cross-section of the template shape, and one or more control points defined on the selected target cross-section that are included in the control point group defined corresponding to the cross-section group for the selected target direction.

[0019] The information processing device 1 accepts a movement operation for one or more control points on a target cross section. The information processing device 1 deforms the template shape according to the arrangement of one or more control points included in each control point group after the movement operation. The acceptance of the movement operation for the control points and the deformation of the template shape may be repeated any number of times. The deformation process of the template shape may be performed after or during the movement operation. In each control point group, a free deformation of the template shape is calculated based on the displacement of each control point. That is, the position of each control point after the movement operation determines the deformation field due to each control point group. The overall deformation field of the template shape is determined by combining the deformation fields due to the displacement of each control point calculated for each control point group. The operator sets each direction as the target direction, sets each cross section included in the cross section group for each direction as the target cross section, and, while checking the corresponding target image, performs a movement operation on the control points set on each target cross section so as to deform the template shape to fit the structure of the object depicted in the target image. Basically, the operator checks each direction in any order and each cross section in each direction multiple times in any order, and performs the movement operation. As a result, the template shape adapts to the structure of the object shown in each target image. The annotation is completed when the template shape adapts to the object shown in each target image in all cross sections in all directions. After the movement operation is completed, the information processing device 1 outputs the deformed template shape as an annotation for the structure of the object shown in two or more target images.

[0020] The two or more directions may not be particularly limited and may be determined appropriately depending on the embodiment. The directions included in the two or more directions may be defined appropriately to face in different directions from each other. The angle between one direction and another direction (a combination of two directions) included in the two or more directions may be an acute angle, a perpendicular angle, or an obtuse angle. As a typical example, the angle between each direction may be perpendicular, like the x-axis, y-axis, and z-axis of a three-dimensional coordinate system. However, the two or more directions are not limited to this example, and the number of directions and the angle between each direction may be selected appropriately depending on the embodiment. The direction may be a straight line or a non-straight line (for example, a line with a complex shape, such as a curved or bent line).

[0021] A group of cross sections and a group of control points are defined for each direction. The number of groups of cross sections and groups of control points corresponds to the number of defined directions. For example, if three directions are defined, three groups of cross sections and three groups of control points are defined. The number of cross sections included in the group of cross sections for each direction is not particularly limited and may be determined appropriately depending on the embodiment. The number of cross sections included in the group of cross sections for each direction may be the same or different. Furthermore, the dimensions and shapes of the cross sections included in the group of cross sections for each direction may be the same or different. The dimensions and shapes of the cross sections within the group of cross sections for one direction may also be the same or different. The cross sections included in each group of cross sections may be perpendicular to the defined direction or may be tilted from the perpendicular. The spacing between the cross sections within the group of cross sections for each direction is not particularly limited and may be determined appropriately depending on the embodiment. These parameters of each group of cross sections are not particularly limited and may be determined appropriately depending on the embodiment.

[0022] The control point group for each direction may be defined so that one or more control points are arranged for each cross section included in the corresponding cross section group. The number of control points included in each control point group for each direction may be determined appropriately depending on the embodiment. At least one of the cross section groups may include a reference cross section, and control points may not be set on this cross section. As a result, there may be cross sections that appear to have no control points arranged on them. One or more control points set on a cross section may also be referred to as a control point set. The number of control point sets included in each control point group corresponds to the number of cross sections included in the corresponding cross section group (excluding the reference cross section, if any). For example, if the number of cross sections included in the corresponding cross section group is four, the number of control point sets included in the control point group is also four. The number of control point sets included in each control point group between directions may be the same or different depending on the cross section group settings. The number and arrangement of control points in the control point set included in each control point group between directions may be the same or different. The number and arrangement of control points included in each control point set within a unidirectional control point group may be the same or different, and these parameters of each control point group are not particularly limited and may be determined appropriately depending on the embodiment.

[0023] Each target image group may be acquired appropriately corresponding to each defined cross-sectional group. In a typical example, an image (target image) may be acquired for each cross-section of each cross-sectional group. However, as long as a target image corresponding to each cross-section can be displayed, the configuration of the two or more target image groups is not limited to this example and may be selected appropriately depending on the embodiment. In another example, the two or more target image groups may be configured using three-dimensional model data that can generate a target image group corresponding to each cross-sectional group (i.e., that can acquire a target image corresponding to each cross-section in each direction). At least one of one or more target images included in at least one of the two or more target image groups may be composed of one or more images. In other words, multiple images that constitute one cross-section may be considered as one target image. Note that parameters of the target image group correspond to parameters of the cross-sectional group. The parameters of the cross-sectional group may be defined as parameters of the target image group. In other words, the parameters of the cross-sectional group may be determined according to the acquired target image group.

[0024] For example, assume that a first direction and a second direction (two directions) are defined as two or more directions, five cross sections are defined in the first direction, eight cross sections are defined in the second direction, nine control points are set on each cross section in the first direction, and four control points are set on each cross section in the second direction. In this case, the total number of cross sections (target images) is "5 + 8 = 13," and the total number of control points is "9 × 5 + 4 × 8 = 77." Note that the number of cross sections in each direction and the number of control points on each cross section may each be given in advance (i.e., may be a fixed value) or may be given dynamically (i.e., may be variable).

[0025] In one example, as illustrated in FIG. 1 , the two or more directions may include three directions: a vertical direction (vertical axis), a left-right direction (left-right axis), and an anterior-posterior direction (anterior-posterior axis). The left-right axis may also be referred to as a coronal horizontal axis or a frontal horizontal axis. The anterior-posterior axis may also be referred to as a sagittal horizontal axis. Accordingly, the two or more cross sections may include a horizontal plane group, a sagittal plane group, and a coronal plane group. The horizontal plane group may also be referred to as a transverse plane group. The coronal plane group may also be referred to as a coronal plane group. The control point group 31 in FIG. 1 is defined corresponding to the horizontal plane group, the control point group 32 is defined corresponding to the sagittal plane group, and the control point group 33 is defined corresponding to the coronal plane group. The horizontal plane group may include one or more horizontal planes 211, and the control point group 31 may include one or more control points 311 for each horizontal plane 211. The sagittal plane group may include one or more sagittal planes 221, and the control point group 32 may include one or more control points 321 for each sagittal plane 221. The coronal plane group may include one or more coronal planes 231, and the control point group 33 may include one or more control points 331 for each coronal plane 231. Note that, although 20 control points 311 are set on the horizontal plane 211 in FIG. 1 , this is merely an example. The number of control points 311 set on the horizontal plane 211 is not limited to this example, and may be determined appropriately depending on the embodiment.

[0026] FIG. 2 schematically illustrates an example of a template shape 5 before deformation, target images (611, 621, 631) of each cross section (horizontal plane 211, sagittal plane 221, coronal plane 231), and a set of control points (311, 321, 331) according to this embodiment. FIG. 3 schematically illustrates an example of a template shape 5 after deformation, target images (611, 621, 631) of each cross section, and a set of control points (311, 321, 331) according to this embodiment. In FIGS. 2 and 3 , object images (711, 721, 731) appearing in the target images (611, 621, 631) are indicated by dotted lines. Note that, in one example, as shown in each figure, the control points (311, 321, 331) may be arranged as grid intersections in each cross section. However, the arrangement of the control points (311, 321, 331) is not limited to this example and may be changed as appropriate depending on the embodiment. Furthermore, a combination of at least two of the horizontal plane 211, the sagittal plane 221, and the coronal plane 231 may be displayed simultaneously on the display 141, or may be displayed separately.

[0027] 2 and 3 , the information processing device 1 may display, for each horizontal plane 211, a corresponding target image 611, one or more control points 311, and a partial shape 511 of the template shape 5. The operator performs a movement operation on the one or more control points 311, thereby deforming the template shape 5, so that the division of the partial shape 511 on the horizontal plane 211 matches the division of the structure appearing in an image 711 of the object depicted in the target image 611. Similarly, the information processing device 1 may display, for each sagittal plane 221, a corresponding target image 621, one or more control points 321, and a partial shape 521 of the template shape 5. The operator performs a movement operation on the one or more control points 321, thereby deforming the template shape 5, so that the division of the partial shape 521 on the sagittal plane 221 matches the division of the structure appearing in an image 721 of the object depicted in the target image 621. For each coronal plane 231, the information processing device 1 may display a corresponding target image 631, one or more control points 331, and a partial shape 531 of the template shape 5. The operator performs a movement operation on the one or more control points 331, thereby deforming the template shape 5, so that the division of the partial shape 531 on the coronal plane 231 matches the division of the structure appearing in an image 731 of the target object captured in the target image 631.

[0028] As shown in FIG. 2 , because the structure of an object may vary from one object to another, accurate annotation is difficult simply by providing the template shape 5. Therefore, the operator checks the vertical, left-right, and front-to-back directions in any order, and in each direction, checks each cross section (horizontal plane 211, sagittal plane 221, coronal plane 231) in any order, and performs the movement operation as described above. The overall deformation field of the template shape 5 is obtained by multiplying the deformation fields of each control point group (31, 32, 33). As the operator moves the template shape 5 in each cross section, it adapts to the structure of the object depicted in each target image (611, 621, 631) included in each target image group. As shown in FIG. 3 , annotation is complete when the partial shapes (511, 521, 531) of the template shape 5 adapt to the object images (711, 721, 731) depicted in each target image (611, 621, 631) in all cross sections in all directions. As shown in Figures 2 and 3, the transformation of this template shape (template shape 5 in the figures) can be performed intuitively by moving the control points to transform the template shape so that it matches the structure of the object depicted in each target image. Therefore, this transformation is easy. However, if a control point group is provided in common in all directions, a control point moved on a cross section in one direction may disappear from the cross section group in that direction by moving away from the cross section in the other direction.

[0029] The disappearance of this control point will be explained using Figures 4A to 4D and Figures 5A to 5D. Figure 4A schematically shows an example of the initial state of the control point group 31 (control points 311) defined for the horizontal plane group. In the example of Figure 4A, four horizontal planes 211 are defined, and 16 control points 311 are defined on each horizontal plane 211. Figure 4B schematically shows an example of a target image 611 on the horizontal plane 211, control points 311 on the horizontal plane 211, a partial shape 511 of the template shape 5, and an image 711 of the object captured in the target image 611, when the control point group 31 is in the initial state. Figure 4C schematically shows an example of a target image 621 on the sagittal plane 221, the arrangement of the control points 311 observed in the sagittal plane 221, the partial shape 521 of the template shape 5, and an image 721 of the object captured in the target image 621, when the control point group 31 is in the initial state. FIG. 4D schematically illustrates an example of a target image 631 in the coronal plane 231, the arrangement of control points 311 observed in the coronal plane 231, a partial shape 531 of the template shape 5, and an image 731 of an object captured in the target image 631, when the control point cloud 31 is in an initial state. FIG. 5A schematically illustrates an example of a movement operation of the control point 311 in the horizontal plane 211 of FIG. 4B. For convenience, in the following description, the control point 311 that has been moved will also be referred to as a "control point 311Z." FIG. 5B schematically illustrates an example of the state of the control point cloud 31 after the movement operation of the control point 311 in FIG. 5A. FIG. 5C schematically illustrates an example of the arrangement of the control point 311 observed in the sagittal plane 221 after the movement operation of the control point 311 in FIG. 5A. FIG. 5D schematically shows an example of the arrangement of the control point 311 observed on the coronal plane 231 after the movement operation of the control point 311 in FIG. 5A.

[0030] In order to explain a situation in which a common control point group is assigned to three directions, namely the vertical direction, the left-right direction, and the front-back direction, it is assumed that the cross-section spacing and the control point spacing in each direction are the same, and that, in the initial state, each control point included in the control point group in each direction is also located on a cross-section in another direction. That is, it is assumed that, in the initial state, each control point 311 of the control point group 31 defined with respect to the horizontal plane group can also be observed in the sagittal plane 221 and the coronal plane 231. However, the cross-section spacing and the control point spacing in each direction are not limited to this example and may be determined appropriately depending on the embodiment.

[0031] As illustrated in Figures 4A to 4D and 5A to 5D, movement operations are accepted on each cross section. Therefore, the control point that has been moved will not be lost on the target cross section, but may be lost on cross sections in other directions. As illustrated in Figure 5A, when a control point 311Z is moved on a horizontal plane 211, the control point 311Z remains observable on the horizontal plane 211 after the movement operation. However, the movement on the horizontal plane 211 changes the position of the control point 311Z in the left-right and front-back directions. Therefore, as illustrated in Figures 5C and 5D, the control point 311Z may be lost (i.e., may no longer be observable) on at least one of the sagittal plane 221 and the coronal plane 231. Losing a control point (i.e., the control point disappearing) in this way may make the work difficult. In other words, the benefits of easy work provided by the control points may not be realized. In contrast, in this embodiment, a control point group is assigned independently for each direction. In the above example, control point groups (31, 32, 33) are independently assigned to the vertical direction, left-right direction, and front-rear direction, respectively.

[0032] 6A schematically illustrates an example of the control point group 32 defined for the sagittal plane group, the target image 621 of the sagittal plane 221, the control points 321 on the sagittal plane 221 included in the control point group 32, the partial shape 521 of the template shape 5, and an image 721 of the object captured in the target image 621 after a movement operation of the control point 311 in FIG. 5A . FIG. 6B schematically illustrates an example of the control point group 33 defined for the coronal plane group, the target image 631 of the coronal plane 231, the control points 331 on the coronal plane 231 included in the control point group 33, the partial shape 531 of the template shape 5, and an image 731 of the object captured in the target image 631 after a movement operation of the control point 311 in FIG. 6A . As illustrated in FIGS. 6A and 6B , the control point group 32 of the sagittal plane group and the control point group 33 of the coronal plane group are independent of the control point group 31 of the horizontal plane group and are therefore not affected by a movement operation of the control point group 31. That is, the control points (321, 331) included in each control point group (32, 33) do not move when the control point 311 in the control point group 31 is moved. The control point 321 in the control point group 32 moves only on the sagittal plane 221, and the control point 331 in the control point group 33 moves only on the coronal plane 231. Therefore, it is possible to keep track of the control points (321, 331) in the sagittal plane 221 and the coronal plane 231. Therefore, in this embodiment, regardless of the direction in which the movement operation is performed, the control points can be prevented from disappearing from the cross section in each direction. Therefore, according to this embodiment, annotations can be added in an easy manner. In one example of the above embodiment, annotations can be added in an easy manner when observing an object in the horizontal plane 211, the sagittal plane 221, and the coronal plane 231.

[0033] (Template Shape) The template shape (template shape 5) is configured to represent the three-dimensional shape of the object and to identify structural divisions within the object. The divisions may be referred to as portions or sections. As long as it is configured in this manner, the data format of the template shape is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the template shape may be given as three-dimensional volume data, polygon data, or the like. The template shape may use a known data format for representing a three-dimensional shape.

[0034] (Object / Object Image) The object may not be particularly limited and may be selected appropriately depending on the embodiment. In one example, the object may include an object whose internal structure is likely to vary from one individual to another, such as biological tissue, a cell group, or an industrial product. The biological tissue may include musculoskeletal tissue, internal organs, blood vessels, or a combination thereof. Furthermore, as long as it is possible to display an image of the object in each cross section, the data format of the object images included in each object image group may not be particularly limited and may be selected appropriately depending on the embodiment.

[0035] In one example, the object may be biological tissue. Each target image group may be any one of a magnetic resonance (MR) image group, a computed tomography (CT) image group, a micro-computed tomography (MCT) image group, an ultrasound image group, and an optical coherence tomography (OCT) image group. According to this example embodiment, annotations can be added easily in a situation where each image group is used to observe biological tissue. Also, in one example, the biological tissue may be musculoskeletal. According to this example embodiment, musculoskeletal annotations can be added easily.

[0036] The number of target cross sections displayed on the display 141 may be one or two or more. The information processing device 1 may simultaneously display multiple cross sections (target image, partial shapes of the template shape, and control point sets) in the same direction on the display 141. Furthermore, as exemplified in FIG. 2 etc., the information processing device 1 may simultaneously display one or more cross sections in each of multiple directions on the display 141. The information processing device 1 may accept movement operations on multiple cross sections in parallel. The display format of the cross sections on the display 141 is not particularly limited and may be determined appropriately depending on the embodiment. The display 141 may be directly connected to the information processing device 1 or indirectly connected via an external computer.

[0037] (Movement Operation) The movement operation is an operation for changing the position of a target control point on the target cross section. The method of this movement operation is not particularly limited and may be designed appropriately depending on the embodiment. In a typical example, the movement operation may be performed by a drag operation using a mouse or a touch panel. The deformation of the template shape is performed by free deformation according to the displacement of each control point included in each control point group. The amount of deformation of the template shape is determined according to the displacement direction and displacement amount of each control point. For this free deformation method, a known method such as that described in the reference (Hsu, W. M., Hughes, J. F. & Kaufman, H. Direct manipulation of free-form deformations. ACM Siggraph Computer Graphics 26, 177-184, (1992)) may be adopted.

[0038] §2 Configuration Example (Hardware Configuration Example) Fig. 7 shows a schematic diagram of an example of the hardware configuration of the information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment is a computer in which a control unit 11, a storage unit 12, an input device 13, an output device 14, and a drive 15 are electrically connected.

[0039] The control unit 11 includes a hardware processor such as a central processing unit (CPU), a random access memory (RAM), and a read-only memory (ROM), and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource.

[0040] The storage unit 12 may be configured, for example, with a hard disk drive, a solid state drive, or the like. The storage unit 12, RAM, and ROM are examples of memory resources. In this embodiment, the storage unit 12 stores various information such as a program 81. The program 81 is a program for causing the information processing device 1 to execute information processing related to the annotation work (see FIG. 9 described below). The program 81 includes a series of instructions for the information processing.

[0041] The input device 13 is a device for inputting, for example, a mouse, a keyboard, an operator, etc. The output device 14 is a device for outputting, for example, a speaker, etc. In this embodiment, the output device 14 may include a display 141. An operator (user) can operate the information processing device 1 by using the input device 13 and the output device 14. The input device 13 and the output device 14 may be connected via an external interface. The input device 13 and the output device 14 (display 141) may be integrally configured, for example, by a touch panel display, etc.

[0042] The drive 15 is a device for reading various information, such as a program, stored in a storage medium 91. The program 81 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store various information (such as the stored program) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The information processing device 1 may obtain the program 81 from the storage medium 91. The storage medium 91 may be a disk-type storage medium such as a CD or DVD, or a non-disk-type storage medium such as a semiconductor memory (e.g., a flash memory). The drive 15 may be connected via an external interface. The external interface is configured to connect to an external device via a wired or wireless connection.

[0043] Note that, with regard to the specific hardware configuration of the information processing device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or the like. The storage unit 12 may be configured with RAM and ROM included in the control unit 11. At least one of the input device 13, the output device 14, and the drive 15 may be omitted. For example, the information processing device 1 may be connected to another computer via an external interface, and input and output may be performed via the other computer. In this case, the output destination of the target image, etc. may be the display of the other computer, and the input device 13 and the output device 14 may be omitted. Note that the external interface may be, for example, a universal serial bus (USB) port, a communication port, a dedicated port, etc. The type and number of external interfaces may be determined as appropriate depending on the embodiment. The information processing device 1 may be an information processing device designed specifically for the service provided, or may be a general-purpose server device, a general-purpose PC (Personal Computer), a tablet PC, a mobile terminal including a smartphone, or the like.

[0044] (Software Configuration Example) Fig. 8 schematically shows an example of the software configuration of the information processing device 1 according to this embodiment. The control unit 11 of the information processing device 1 loads a program 81 stored in the storage unit 12 into RAM and executes instructions included in the program 81 using the CPU. As a result, the information processing device 1 operates as a computer including an acquisition unit 111, a display processing unit 112, an operation acceptance unit 113, a transformation processing unit 114, and an output processing unit 115 as software modules. That is, in this embodiment, each software module of the information processing device 1 is realized by the control unit 11 (CPU).

[0045] The acquisition unit 111 is configured to acquire two or more target image groups of the object. Each of the two or more target image groups corresponds to one of two or more defined cross-section groups. The display processing unit 112 is configured to display, on the display 141, a corresponding target image included in the acquired two or more target image groups, a template shape indicating a division of the object's structure, and one or more corresponding control points included in the corresponding control point group for one or more target cross-sections included in the cross-section group defined for a target direction out of two or more directions. The operation receiving unit 113 is configured to receive a movement operation for one or more control points in the target cross-section. The transformation processing unit 114 is configured to transform the template shape according to the arrangement of one or more control points included in each control point group after the movement operation. The output processing unit 115 is configured to output the transformed template shape as an annotation for the object's structure shown in the two or more target image groups after the movement operation is completed.

[0046] In this embodiment, an example is described in which each software module of the information processing device 1 is implemented by a general-purpose CPU. However, some or all of the software modules may be implemented by one or more dedicated processors or chipsets. Each module may be implemented as a hardware module. With regard to the software configuration of the information processing device 1, modules may be omitted, replaced, or added as appropriate.

[0047] §3 Operational Example Figure 9 is a flowchart showing an example of the processing procedure of the information processing device 1 according to this embodiment. The following processing procedure is an example of an information processing method executed by a computer. However, the following processing procedure is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0048] In step S101, the control unit 11 operates as the acquisition unit 111 and acquires two or more target image groups of the object. In one example, the control unit 11 may acquire three target image groups: a target image group corresponding to a horizontal plane group, a target image group corresponding to a sagittal plane group, and a target image group corresponding to a coronal plane group. The information source is not particularly limited and may be determined appropriately depending on the embodiment. In one example, the control unit 11 may acquire two or more target image groups by reading data stored in the storage unit 12 or the storage medium 91. Furthermore, the control unit 11 may acquire two or more target image groups from a device that generates the target image groups or an external computer. The device that generates the target image groups may be, for example, an MRI (Magnetic Resonance Imaging) device, a CT device, a micro-CT device, an ultrasound diagnostic device, an optical coherence tomography device, or the like. After acquiring two or more target image groups, the control unit 11 proceeds to the next step S102.

[0049] In step S102, the control unit 11 operates as the display processing unit 112 and displays, for a target cross section, a corresponding target image, a corresponding partial shape of the template shape, and one or more corresponding control points (control point sets) on the display 141. The target cross section to be displayed may be specified by an operator or may be automatically selected by a computer. Furthermore, in step S102, the selection and display of the target cross section may be repeatedly executed. By outputting each piece of information related to the target cross section, the control unit 11 may start accepting a movement operation for the control points on the target cross section. After outputting each piece of information related to the target cross section, the control unit 11 proceeds to the next step S103.

[0050] In step S103, the control unit 11 determines whether or not to terminate the acceptance of the movement operation for the control point. The criteria for the determination may be set arbitrarily. In one example, the control unit 11 may determine whether or not to terminate the acceptance of the movement operation depending on whether or not a predetermined termination operation (such as pressing an end button) has been performed. If it is determined that the acceptance of the movement operation should not be terminated, the control unit 11 proceeds to the next step S104. If it is determined that the acceptance of the movement operation should be terminated, the control unit 11 proceeds to step S106.

[0051] In step S104, the control unit 11 operates as the operation receiving unit 113 and receives a movement operation for one or more control points on the target cross section. In step S105, the control unit 11 operates as the deformation processing unit 114 and deforms the template shape according to the arrangement of one or more control points included in each control point group after the movement operation. The control unit 11 may repeatedly execute the processes of steps S104 and S105 on the same target cross section. When the deformation of the template shape is completed, the control unit 11 returns the process to step S102 and repeats the process from step S102.

[0052] When returning to step S102, the control unit 11 may accept a change in the target direction and the target cross section, or only the target cross section, from the operator. The control unit 11 may execute each process from step S102 onwards for the changed target cross section. Furthermore, the control unit 11 may accept an end operation at any timing. In response to this, the control unit 11 may proceed to step S106.

[0053] In step S106, the control unit 11 operates as the output processing unit 115 and, after the movement operation is completed, outputs the deformed template shape as an annotation for the structure of the object appearing in two or more target image groups. The output format of the template shape is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the control unit 11 may output the deformed template shape directly to the output device 14 (such as the display 141) or an output device of an external computer. In another example, outputting the deformed template shape may include saving the deformed template shape in a predetermined storage area. The predetermined storage area is not particularly limited and may be selected appropriately depending on the embodiment. The predetermined storage area may be, for example, RAM within the control unit 11, the storage unit 12, an external storage device, a storage medium, or a combination thereof. The storage medium may be, for example, a CD, a DVD, a semiconductor memory, or the like, and the control unit 11 may store the template shape data in the storage medium via the drive 15. The external storage device may include a storage device of an external computer. The external storage device may be, for example, a data server such as a NAS. In this case, the control unit 11 may use an external interface (communication port) to store data of the deformed template shape in a data server via a network. The external storage device may be, for example, an external storage device. The external storage device may be appropriately connected to the information processing device 1. For example, the information processing device 1 may be connected to an external storage device via the external interface, and the data of the deformed template shape may be stored in the connected external storage device. The control unit 11 may save the generated deformed template shape in association with two or more acquired target image groups. When the output of the template shape is complete, the control unit 11 terminates the processing procedure of the information processing device 1 according to this operation example.

[0054] [Features] In this embodiment, steps S104 and S105 accept a deformation operation of the template shape by moving each control point on each cross section in each direction as an annotation operation for the structure of the object. This deformation operation can be performed intuitively by moving the control points to deform the template shape so that it matches the structure of the object depicted in the target image of the target cross section, as exemplified in FIG. 5A and other figures. Therefore, this deformation operation is easy. Furthermore, in this embodiment, a control point group is independently assigned for each direction, thereby preventing the control points from disappearing from the cross sections in each direction. In other words, it is possible to prevent the control points from being lost on any cross section while repeating the processing of steps S104 and S105. Therefore, this embodiment allows annotation to be added in a simple manner.

[0055] §4 Modifications Although the embodiments of the present disclosure have been described in detail above, the above description is merely an example of the present disclosure in every respect. It goes without saying that various improvements or modifications can be made without departing from the scope of the present disclosure. The processes and means described in the present disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0056] The series of processes for moving control points and deforming the template shape at each cross section in the above embodiment may be used for purposes other than annotation. In one example, the series of processes for deforming the template shape may be used for deforming a three-dimensional model. Accordingly, an object of the present disclosure may be to deform a three-dimensional model in an easy manner. In this case, the acquisition of two or more target image groups and the display of the target images may be omitted. Alternatively, the acquisition of two or more target image groups and the display of the target images may be performed to show a sample of the deformed three-dimensional model. The template shape may indicate a structural division of the object, or may indicate any other information. In this modification, the template shape corresponds to a three-dimensional model. The template shape may be provided as appropriate depending on the embodiment. The deformed template shape may be output for purposes other than annotation. According to this modification, a three-dimensional model can be deformed in an easy manner. This modification may be used, for example, to deform the shape of clothing (sleeves, torso, etc.) to fit a target person.

[0057] REFERENCE SIGNS LIST 1... information processing device, 211... horizontal plane, 221... sagittal plane, 231... coronal plane, 31, 32, 33... control point group, 311, 321, 331... control point, 5... template shape, 611, 621, 631... target image

Claims

1. A program for causing a computer to execute an information processing method, comprising: two or more cross-section groups each defined for two or more directions; two or more control point groups each independently defined corresponding to each of the two or more cross-section groups; each of the cross-section groups includes one or more cross sections in a defined direction; and each of the control point groups includes one or more control points arranged for each of the cross sections included in the corresponding cross-section group; the information processing method comprises the steps of: acquiring two or more target image groups of an object, each of the two or more target image groups corresponding to each of the two or more defined cross-section groups; displaying, on a display, a corresponding target image included in the acquired two or more target image groups, a template shape indicating a division of the structure of the object, and one or more corresponding control points included in the corresponding control point group for a target cross-section among the one or more cross-sections included in the cross-section group defined for a target direction among the two or more directions; accepting a movement operation for the one or more control points in the target cross-section; deforming the template shape in accordance with an arrangement of the one or more control points included in each control point group after the movement operation; after the moving operation is completed, outputting the deformed template shape as an annotation for a structure of the object captured in the two or more target images.

2. The program of claim 1, wherein the object is biological tissue, and each of the object image groups is one of a magnetic resonance image group, a computed tomography image group, a micro-computed tomography image group, an ultrasound image group, and an optical coherence tomography image group.

3. The program of claim 2, wherein the biological tissue is musculoskeletal.

4. The program according to any one of claims 1 to 3, wherein the two or more cross-section groups include a horizontal plane group, a sagittal plane group, and a coronal plane group.

5. An information processing method executed by a computer, comprising: two or more cross-section groups each defined for two or more directions; two or more control point groups each independently defined corresponding to each of the two or more cross-section groups; each of the cross-section groups includes one or more cross sections in a defined direction; and each of the control point groups includes one or more control points arranged for each of the cross sections included in the corresponding cross-section group; the information processing method comprises the steps of: acquiring two or more target image groups of an object, each of the two or more target image groups corresponding to each of the two or more defined cross-section groups; displaying, on a display, a corresponding target image included in the acquired two or more target image groups, a template shape indicating a division of the structure of the object, and one or more corresponding control points included in the corresponding control point group for a target cross-section among the one or more cross-sections included in the cross-section group defined for a target direction among the two or more directions; accepting a movement operation for the one or more control points in the target cross-section; deforming the template shape in accordance with the arrangement of the one or more control points after the movement operation; After the moving operation is completed, outputting the deformed template shape as an annotation for a structure of the object captured in the two or more target images.

Images