Program and information processing method

Abstract

To provide a technique for giving annotation with an easy method.SOLUTION: A program relating to one aspect of the present invention causes a computer to execute the steps of: acquiring two or more target image groups as to an object; displaying, as to a target cross section, corresponding target images included on the two or more target image groups, a template shape, and one or more corresponding control points; receiving an operation for movement of the one or more displayed control points; deforming the template shape in response to locations of the control points of each of the control point groups obtained after the movement operation; and outputting, as annotation for a structure of the object, the deformed template shape after completion of the movement operation.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a program and an information processing method.

Background Art

[0002] Since muscle mass is an important determinant of the ability to generate force, the quantification of individual muscle mass in the living body has attracted high interest in various fields such as health, medicine, and sports. Muscle mass is evaluated by segmenting muscles from magnetic resonance (MR) images, computed tomography (CT) images, etc. Conventionally, segmentation has been performed manually by experts. However, the task of tracing the contours of each muscle for each slice is very time-consuming, especially when the number of muscles is large. Therefore, in recent years, techniques for automating the segmentation task using a trained image analysis model generated by machine learning have emerged. This artificial intelligence-based segmentation method is very promising.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When adopting an artificial intelligence-based segmentation method, in order to newly train a model, it is necessary to generate a data set composed of a combination of training data and correct labels. For example, Patent Document 1 proposes a system for annotating medical images by inputting a touch panel. However, in such an annotation method, the operator requires specialized knowledge and it is difficult and time-consuming to correctly trace the skeletal muscles. Note that such problems do not occur only in the scenario of annotating medical images. Such problems can occur in various scenarios where annotations are given to the structure of an object.

[0005] In one aspect, the present disclosure has been made in view of such circumstances, and its purpose is to provide a technique for giving annotations in an easy way.

Means for Solving the Problems

[0006] In order to solve the above-described problems, the present disclosure adopts the following configurations. Note that the following configurations can be combined as appropriate.

[0007] A program according to an aspect of the present disclosure is a program for causing a computer to execute an information processing method. Each of two or more cross-section groups is defined for each of two or more directions. Each of two or more control point groups is independently defined corresponding to each of the two or more cross-section groups. Each of the cross-section groups includes one or more cross-sections in the defined direction. Each of the control point groups includes one or more control points arranged for each of the cross-sections included in the corresponding cross-section group. The information processing method includes a step of acquiring two or more target image groups for an object, wherein each of the two or more target image groups corresponds to each of the two or more defined cross-section groups, a step of, for a target cross-section among the one or more cross-sections included in the cross-section group defined for a target direction among the two or more directions, displaying, on a display, a corresponding target image included in the two or more acquired target image groups, a template shape indicating a classification of the structure of the object, and one or more corresponding control points included in the corresponding control point group, a step of receiving a movement operation on the one or more control points in the target cross-section, after the movement operation deforming the template shape according to the arrangement of the one or more control points included in each of the control point groups, and after the movement operation is completed, outputting the deformed template shape as an annotation for the structure of the object shown in the two or more target image groups.

[0008] In this configuration, a template shape is given for the classification of the structure of the object. However, since there can be individual differences in the structure of the object, it is difficult to perform accurate annotation simply by giving the template shape. Therefore, cross-sectional groups are defined in two or more directions, and as an annotation operation for the structure of the object, a deformation operation of the template shape by moving each control point in each cross-section in each direction is accepted. This deformation operation can be performed by an intuitive operation of moving the control points so that the template shape is deformed to match the structure of the object shown in the target image of the target cross-section. Therefore, this deformation operation is easy. However, if the control point group is given in common in all directions, the control points moved on the cross-section in one direction may move away from the cross-sections in other directions and disappear from the cross-sectional groups in those directions. When the control points disappear, the work may become difficult (that is, the benefit of the easy work may not be received). In contrast, in this configuration, by giving the control point group independently in each direction, the disappearance of the control points from the cross-sections in each direction can be prevented. Therefore, according to this configuration, annotation can be given in an easy way.

[0009] In the program according to the above aspect, the object may be a biological tissue. Each of the target image groups may be any one of a magnetic resonance image group, a computed tomography image group, a micro-computed tomography image group, an ultrasonic image group, and an optical coherence tomography image group. According to this configuration, in a scene where each image group is used for observation of a biological tissue, annotation can be given in an easy way.

[0010] In the program according to the above aspect, the biological tissue may be a musculoskeletal system. According to this configuration, annotation of the musculoskeletal system can be given in an easy way.

[0011] In the program according to the above aspect, the two or more cross-sectional groups may include a horizontal plane group, a sagittal plane group, and a coronal plane group. According to this configuration, in a scene where the object is observed in the horizontal plane, sagittal plane, and coronal plane, annotation can be given in an easy way.

[0012] Note that the embodiments of the present disclosure are not necessarily limited to the above program. As another aspect of the program according to each of the above aspects, one aspect of the present disclosure may be an information processing apparatus that realizes all or a part of each of the above configurations, may be an information processing method, or may be a machine-readable storage medium such as a computer that stores the above program. A machine-readable storage medium such as a computer is a medium that stores information such as a program by an electrical, magnetic, optical, mechanical, or chemical action.

[0013] For example, the information processing method according to one aspect of the present disclosure may be an information processing method executed by a computer. The information processing method includes a step of acquiring two or more target image groups for a target object, wherein each of the two or more target image groups corresponds to each of the defined two or more cross-section groups, a step of, for a target cross-section among the one or more cross-sections included in the cross-section group defined for a target direction among the two or more directions, displaying, on a display, a corresponding target image included in the two or more target image groups acquired, a corresponding one or more control points included in a template shape indicating a classification of the structure of the target object and a corresponding control point group, a step of receiving a movement operation on the one or more control points in the target cross-section, a step of deforming the template shape according to the arrangement of the one or more control points after the movement operation, and a step of outputting, as an annotation for the structure of the target object shown in the two or more target image groups after the movement operation is completed, the deformed template shape. The step may include applying force.

Advantages of the Invention

[0014] According to the present disclosure, annotation can be given in an easy manner.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 6A

Figure 6B

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0016] Hereinafter, embodiments according to one aspect of the present disclosure (hereinafter also referred to as "the present embodiment") will be described with reference to the drawings. However, the present embodiment described below is merely an exemplification of the present disclosure in every respect. Needless to say, various improvements and modifications can be made without departing from the scope of the present disclosure. That is, in implementing the present disclosure, a specific configuration according to the embodiment may be appropriately adopted. Note that the data appearing in the present embodiment is described in natural language, but more specifically, it is specified by a quasi-language, command, parameter, machine language, etc. that can be recognized by a computer.

[0017] §1 Application Example FIG. 1 schematically shows an example of a scene to which the present disclosure is applied. The information processing apparatus 1 according to the present embodiment is one or more computers configured to receive an annotation operation on the structure of an object. In the present embodiment, each of two or more cross-section groups is defined for each of two or more directions. Each of two or more control point groups is independently defined corresponding to each of the two or more cross-section groups. Each cross-section group includes one or more cross-sections in the defined direction. Each control point group includes one or more control points arranged for each cross-section included in the corresponding cross-section group.

[0018] The information processing apparatus 1 is connected to the display 141. The information processing apparatus 1 acquires two or more groups of target images for the object. Each of the two or more groups of target images corresponds to each of the two or more defined cross-section groups. The information processing apparatus 1 displays, on the display 141, the corresponding target images included in the two or more acquired groups of target images, the template shape indicating the classification of the structure of the object, and the corresponding one or more control points included in the corresponding control point group, for the target cross-section among the one or more cross-sections included in the cross-section group defined for the target direction among the two or more directions. In one example, the information processing apparatus 1 selects one direction as the target direction from the two or more directions. The information processing apparatus 1 selects one cross-section as the target cross-section from the cross-section group defined for the selected target direction. Then, the information processing apparatus 1 displays, on the display 141, the target image corresponding to the selected target cross-section, the partial shape (i.e., the cross-sectional shape) on the target cross-section of the template shape, and the one or more control points defined on the selected target cross-section and included in the control point group defined corresponding to the cross-section group of the selected target direction.

[0019] The information processing apparatus 1 receives a movement operation for one or more control points in a target cross-section. The information processing apparatus 1 deforms the template shape according to the arrangement of one or more control points included in each control point group after the movement operation. The reception of the movement operation for the control points and the deformation of the template shape may be repeatedly executed any number of times. The deformation process of the template shape may be executed after the movement operation or during the movement operation. In each control point group, a free deformation of the template shape is calculated based on the displacement of each control point. That is, the deformation field by each control point group is determined by the position of each control point after the movement operation. The overall deformation field of the template shape is determined by synthesizing the deformation fields by the displacements of each control point calculated by each control point group. The operator sets each direction as the target direction and each cross-section included in the cross-section group in each direction as the target cross-section, and performs a movement operation on the control points set for each target cross-section so as to deform the template shape to conform to the structure of the object shown in the corresponding target image while checking the corresponding target image. Basically, the operator checks each cross-section in each direction in an arbitrary order a plurality of times and performs a movement operation. As a result, the template shape comes to conform to the structure of the object shown in each target image. Annotation is completed at the stage where the template shape conforms to the object shown in each target image for all cross-sections in all directions. After the movement operation is completed, the information processing apparatus 1 outputs the deformed template shape as an annotation for the structure of the object shown in two or more target image groups.

[0020] The two or more directions do not have to be particularly limited and may be appropriately determined according to the embodiment. Each direction included in the two or more directions may be appropriately defined so as to face different directions. The angle between one direction and another direction (a combination of two directions) included in the two or more directions may be an acute angle, a right angle, or an obtuse angle. As a typical example, the angles between the respective directions may be perpendicular, such as the x-axis, y-axis, and z-axis of a three-dimensional coordinate system. However, the two or more directions do not have to be limited to such an example, and the number of directions and the angle between each direction may be appropriately selected according to the embodiment. The direction may be a straight line or not (for example, a line with a complex shape such as curved or bent).

[0021] A cross-section group and a control point group are defined for each direction. The number of the cross-section group and the control point group corresponds to the number of defined directions. For example, when three directions are defined, three cross-section groups and three control point groups are defined. The number of cross-sections included in the cross-section group of each direction does not have to be particularly limited and may be appropriately determined according to the embodiment. Between each direction, the number of cross-sections included in the cross-section group may be the same or different. Also, between each direction, the dimensions and shapes of the cross-sections included in the cross-section group may be the same or different respectively. Between each cross-section within the cross-section group of one direction, the dimensions and shapes may also be the same or different respectively. Each cross-section included in each cross-section group may be perpendicular to the defined direction or may be inclined from the perpendicular. The cross-section interval within the cross-section group of each direction does not have to be particularly limited and may be appropriately determined according to the embodiment. These parameters of each cross-section group do not have to be particularly limited and may be appropriately determined according to the embodiment.

[0022] The control point groups in each direction may be defined such that one or more control points are arranged for each cross section included in the corresponding cross section group. The number of control points included in the control point groups in each direction may be appropriately determined according to the embodiment. Note that at least any one of the cross section groups may include a reference cross section, and no control point may be set for this cross section. As a result, there may seemingly be a cross section where no control point is arranged. One or more control points set for a cross section may also be referred to as a control point set. The number of control point sets included in each control point group corresponds to the number of cross sections included in the corresponding cross section group (excluding the reference cross section if it exists). For example, when the number of cross sections included in the corresponding cross section group is four, the number of control point sets included in the control point group is also four. Between each direction, the number of control point sets included in each control point group may be the same or different corresponding to the setting of the cross section group. Between each direction, the number of control points and the arrangement within each control point set included in each control point group may be the same or different, respectively. Between each control point set within the control point group in one direction, the number of control points included and the arrangement may also be the same or different, respectively. These parameters of each control point group need not be particularly limited and may be appropriately determined according to the embodiment.

[0023] Each target image group may be appropriately acquired corresponding to each defined cross-section group. In a typical example, an image (target image) may be acquired for each cross-section of each cross-section group. However, if it is possible to display the target images corresponding to each cross-section, the configuration of two or more target image groups is not limited to such an example and may be appropriately selected according to the embodiment. In another example, two or more target image groups may be constituted by three-dimensional model data capable of generating target image groups corresponding to each cross-section group (that is, capable of acquiring target images corresponding to each cross-section in each direction). At least one of the one or more target images included in at least any one of the two or more target image groups may be constituted by one or more images. That is, a plurality of images constituting one cross-section may be regarded as one target image. Note that the parameters of the target image group correspond to the parameters of the above cross-section group. The parameters of the above cross-section group may be defined as the parameters of the target image group. That is, according to the acquired target image group, the parameters of the above cross-section group may be determined.

[0024] For example, assume that a first direction and a second direction (two directions) are defined as two or more directions, five cross-sections are defined in the first direction, eight cross-sections are defined in the second direction, nine control points are set for each cross-section in the first direction, and four control points are set for each cross-section in the second direction. In this case, the total number of cross-sections (target images) is "5 + 8 = 13 sheets", and the total number of control points is "9×5 + 4×8 = 77 pieces". Note that the number of cross-sections in each direction and the number of control points in each cross-section may be given in advance (that is, may be fixed values) or may be given dynamically (that is, may be variable).

[0025] In one example, as illustrated in FIG. 1, the two or more directions may include three directions: the vertical direction (vertical axis), the left-right direction (left-right axis), and the front-back direction (front-back axis). The left-right axis may also be referred to as the coronal horizontal axis or the frontal horizontal axis. The front-back axis may also be referred to as the sagittal horizontal axis. Accordingly, the two or more cross-section groups may include a horizontal plane group, a sagittal plane group, and a coronal plane group. The horizontal plane group may be referred to as the transverse section group. The coronal plane group may be referred to as the frontal plane group. The control point group 31 in FIG. 1 is defined corresponding to the horizontal plane group, the control point group 32 is defined corresponding to the sagittal plane group, and the control point group 33 is defined corresponding to the coronal plane group. Horizontal The plane group may include one or more horizontal planes 211, and the control point group 31 may include one or more control points 311 for each horizontal plane 211. The sagittal plane group may include one or more sagittal planes 221, and the control point group 32 may include one or more control points 321 for each sagittal plane 221. The coronal plane group may include one or more coronal planes 231, and the control point group 33 may include one or more control points 331 for each coronal plane 231. In FIG. 1, 20 control points 311 are set on the horizontal plane 211, but this is merely an example. The number of control points 311 provided on the horizontal plane 211 is not limited to such an example and may be appropriately determined according to the embodiment.

[0026] FIG. 2 schematically shows an example of a template shape 5 before deformation, target images (611, 621, 631) of each cross-section (horizontal plane 211, sagittal plane 221, coronal plane 231), and a set of control points (311, 321, 331) according to this embodiment. FIG. 3 schematically shows an example of a template shape 5 after deformation, target images (611, 621, 631) of each cross-section, and a set of control points (311, 321, 331) according to this embodiment. In FIGS. 2 and 3, images (711, 721, 731) of objects shown in each target image (611, 621, 631) are indicated by dotted lines. In an example, as shown in each figure, in each cross-section, each control point (311, 321, 331) may be arranged as an intersection of a grid. However, the arrangement of each control point (311, 321, 331) is not limited to such an example and may be appropriately changed according to the embodiment. Also, combinations of at least two of the horizontal plane 211, sagittal plane 221, and coronal plane 231 may be simultaneously displayed on the display 141, or may be displayed separately.

[0027] As illustrated in FIGS. 2 and 3, the information processing apparatus 1 may display, for each horizontal plane 211, a corresponding target image 611, one or more control points 311, and a partial shape 511 of the template shape 5. The operator performs a movement operation on one or more control points 311 so that the division of the partial shape 511 on this horizontal plane 211 conforms to the division of the structure represented by the image 711 of the object imaged in the target image 611, and deforms the template shape 5. Similarly, the information processing apparatus 1 may display, for each sagittal plane 221, a corresponding target image 621, one or more control points 321, and a partial shape 521 of the template shape 5. The operator performs a movement operation on one or more control points 321 so that the division of the partial shape 521 on this sagittal plane 221 conforms to the division of the structure represented by the image 721 of the object imaged in the target image 621, and deforms the template shape 5. The information processing apparatus 1 may display, for each coronal plane 231, a corresponding target image 631, one or more control points 331, and a partial shape 531 of the template shape 5. The operator performs a movement operation on one or more control points 331 so that the division of the partial shape 531 on this coronal plane 231 conforms to the division of the structure represented by the image 731 of the object imaged in the target image 631, and deforms the template shape 5.

[0028] As shown in FIG. 2, since there can be individual differences in the structure of the object, it is difficult to perform accurate annotation only by providing the template shape 5. Therefore, the operator checks each cross-section (horizontal plane 211, sagittal plane 221, coronal plane 231) in any order in the vertical, left-right, and front-back directions, and performs the movement operation as described above. The overall deformation field of the template shape 5 is obtained by multiplying the deformation fields by each control point group (31, 32, 33). By the movement operation of the operator in each cross-section, the template shape 5 conforms to the structure of the object shown in each target image (611, 621, 631) included in each target image group. As shown in FIG. 3, annotation is completed at the stage where the partial shapes (511, 521, 531) of the template shape 5 conform to the images (711, 721, 731) of the object shown in each target image (611, 621, 631) in all cross-sections in all directions. As illustrated in FIGS. 2 and 3, the deformation operation of this template shape (template shape 5 in the figure) can be performed by an intuitive operation of moving the control points to deform the template shape so as to match the structure of the object shown in each target image. Therefore, this deformation operation is easy. However, if the control point group is provided in common in all directions, the control points moved on a cross-section in one direction may disappear from the cross-section group in that direction by moving away from the cross-sections in other directions.

[0029] Using FIGS. 4A to 4D and FIGS. 5A to 5D, the disappearance of this control point will be described. FIG. 4A schematically shows an example of the initial state of the control point group 31 (control points 311) defined for the horizontal plane group. In the example of FIG. 4A, four horizontal planes 211 are defined, and 16 control points 311 are defined on each horizontal plane 211. FIG. 4B schematically shows an example of the target image 611 of the horizontal plane 211, the control points 311 on the horizontal plane 211, the partial shape 511 of the template shape 5, and the image 711 of the object shown in the target image 611 at the stage where the control point group 31 is in the initial state. FIG. 4C schematically shows an example of the target image 621 of the sagittal plane 221, the arrangement of the control points 311 observed in the sagittal plane 221, the partial shape 521 of the template shape 5, and the image 721 of the object shown in the target image 621 at the stage where the control point group 31 is in the initial state. FIG. 4D schematically shows an example of the target image 631 of the coronal plane 231, the arrangement of the control points 311 observed in the coronal plane 231, the partial shape 531 of the template shape 5, and the image 731 of the object shown in the target image 631 at the stage where the control point group 31 is in the initial state. FIG. 5A schematically shows an example of the movement operation of the control points 311 on the horizontal plane 211 in FIG. 4B. In the following description, for convenience, the control point 311 on which the movement operation is performed is also referred to as "control point 311Z". FIG. 5B schematically shows an example of the state of the control point group 31 after the movement operation of the control points 311 in FIG. 5A. FIG. 5C schematically shows an example of the arrangement of the control points 311 observed in the sagittal plane 221 after the movement operation of the control points 311 in FIG. 5A. FIG. 5D schematically shows an example of the arrangement of the control points 311 observed in the coronal plane 231 after the movement operation of the control points 311 in FIG. 5A.

[0030] Note that, in order to describe a scene in which a common control point group is given in three directions of the vertical direction, the left-right direction, and the front-back direction, the cross-section intervals and the control point intervals in each direction are the same. In the initial state, it is assumed that each control point included in the control point group in each direction is also arranged on the cross-sections in other directions. That is, in the initial state, it is assumed that each control point 311 of the control point group 31 defined for the horizontal plane group is also observable in the sagittal plane 221 and the coronal plane 231. However, the cross-section intervals and the control point intervals in each direction are not limited to such an example and may be appropriately determined according to the embodiment.

[0031] As illustrated in FIGS. 4A to 4D and FIGS. 5A to 5D, the movement operation is received on each cross-section. Therefore, the control point where the movement operation is performed is not lost in the target cross-section, but may be lost in the cross-sections in other directions. As illustrated in FIG. 5A, when the control point 311Z is moved on the horizontal plane 211, the control point 311Z is observed on the horizontal plane 211 even after the movement operation. On the other hand, due to the movement on the horizontal plane 211, the positions of the control point 311Z in the left-right direction and the front-back direction change. Therefore, as illustrated in FIGS. 5C and 5D, this control point 311Z may be lost (i.e., not observed) in at least one of the sagittal plane 221 and the coronal plane 231. When the control point is thus lost (i.e., the control point disappears), the work may become difficult. That is, the benefit of easy work by the control point may not be received. In contrast, in the present embodiment, the control point groups are given independently in each direction. In the above example, control point groups (31, 32, 33) are independently given in the vertical direction, the left-right direction, and the front-back direction, respectively.

[0032] FIG. 6A schematically shows an example of a control point group 32 defined for a sagittal plane group, a target image 621 of the sagittal plane 221, a control point 321 on the sagittal plane 221 included in the control point group 32, a partial shape 521 of the template shape 5, and an image 721 of an object shown in the target image 621, after the movement operation of the control point 311 in FIG. 5A. FIG. 6B schematically shows an example of a control point group 33 defined for a coronal plane group, a target image 631 of the coronal plane 231, a control point 331 on the coronal plane 231 included in the control point group 33, a partial shape 531 of the template shape 5, and an image 731 of an object shown in the target image 631, after the movement operation of the control point 311 in FIG. 5A. As illustrated in FIGS. 6A and 6B, the control point group 32 of the sagittal plane group and the control point group 33 of the coronal plane group are of the horizontal plane group Since it is independent of the control point group 31, it is not affected by the movement operation in the control point group 31. That is, each control point (321, 331) included in each control point group (32, 33) does not move by the movement operation of the control point 311 in the control point group 31. The control point 321 of the control point group 32 moves only on the sagittal plane 221, and the control point 331 of the control point group 33 moves only on the coronal plane 231. Therefore, it is possible to avoid losing each control point (321, 331) on the sagittal plane 221 and the coronal plane 231. Thus, in the present embodiment, no matter in which direction the movement operation is performed, it is possible to prevent the disappearance of the control points from each cross-sectional view. Therefore, according to the present embodiment, annotation can be given in an easy way. In one example of the above-described present embodiment, in a scene where the object is observed on the horizontal plane 211, the sagittal plane 221, and the coronal plane 231, annotation can be given in an easy way.

[0033] (Template shape) The template shape (the template shape is 5) represents the three-dimensional shape of the object and is configured to be able to identify the division of the structure within the object. The division may be referred to as a part or a compartment. If configured in this way, the data format of the template shape does not need to be particularly limited and may be appropriately selected according to the embodiment. In one example, the template shape may be given as three-dimensional volume data, polygon data, or the like. A known data format for representing a three-dimensional shape may be used for the template shape.

[0034] (Object / Object image) The object does not need to be particularly limited and may be appropriately selected according to the embodiment. In one example, the object may include an object in which individual differences are likely to occur in the internal structure of a biological tissue, a cell group, an industrial product, or the like. The biological tissue may include a musculoskeletal system, internal organs, blood vessels, or a combination thereof. Also, if it is possible to display the image of the object in each cross section, the data format of the object images included in each object image group does not need to be particularly limited and may be appropriately selected according to the embodiment.

[0035] In one example, the object may be a biological tissue. Each group of target images may be any of a magnetic resonance (MR) image group, a computed tomography (CT) image group , a micro-computed tomography image group, an ultrasonic image group, and an optical coherence tomography (OCT) image group. According to an example of the present embodiment, in a scenario where each image group is used for observing a biological tissue, annotation can be given in an easy manner. Also, in one example, the biological tissue may be a musculoskeletal system. According to an example of the present embodiment, annotation of the musculoskeletal system can be given in an easy manner.

[0036] Note that the number of target cross-sections to be displayed on the display 141 may be one or two or more. The information processing apparatus 1 may simultaneously display a plurality of cross-sections (target images, partial shapes of the template shape, and control point sets) in the same direction on the display 141. Also, as illustrated in FIG. 2 and the like, the information processing apparatus 1 may simultaneously display one or more cross-sections in each of a plurality of directions on the display 141. The information processing apparatus 1 may accept a movement operation in parallel for a plurality of cross-sections. The display form of the cross-section on the display 141 may not be particularly limited and may be appropriately determined according to the embodiment. The display 141 may be directly connected to the information processing apparatus 1 or may be indirectly connected via an external computer.

[0037] (Movement operation) The movement operation is an operation for changing the position of the control point of the object on the target cross-section. The method of this movement operation may not be particularly limited and may be appropriately designed according to the embodiment. In a typical example, the movement operation may be performed by a drag operation using a mouse or a touch panel. The deformation of the template shape is configured by free deformation according to the displacement of each control point included in each control point group. The deformation amount of the template shape is determined according to the displacement direction and displacement amount of each control point. Examples of this free deformation method include the reference (Hsu, W. M., Hughes, J. F. & Ka Known methods such as Ufman, H. Direct manipulation of free-form deformations. ACM Siggraph Computer Graphics 26, 177-184, (1992). may be adopted.

[0038] §2 Configuration Example (Hardware Configuration Example) FIG. 7 schematically shows an example of the hardware configuration of the information processing apparatus 1 according to the present embodiment. The information processing apparatus 1 according to the present embodiment is a computer in which a control unit 11, a storage unit 12, an input device 13, an output device 14, and a drive 15 are electrically connected.

[0039] The control unit 11 is a CPU (Central Processing Unit) which is a hardware processor, includes a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource.

[0040] The storage unit 12 may be configured by, for example, a hard disk drive, a solid state drive, etc. The storage unit 12, the RAM, and the ROM are examples of memory resources. In the present embodiment, the storage unit 12 stores various information such as a program 81. The program 81 is a program for causing the information processing apparatus 1 to execute information processing related to the above annotation work (FIG. 9 described later). The program 81 includes a series of instructions for the information processing.

[0041] The input device 13 is a device for performing inputs such as a mouse, a keyboard, an operator, etc. The output device 14 is a device for performing outputs such as a speaker, etc. In the present embodiment, the output device 14 may include a display 141. The operator (user) can operate the information processing device 1 by using the input device 13 and the output device 14. The input device 13 and the output device 14 may be connected via an external interface. The input device 13 and the output device 14 (display 141) may be integrally configured by, for example, a touch panel display or the like.

[0042] The drive 15 is a device for reading various information such as programs stored in the storage medium 91. The above program 81 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store the various information (stored programs, etc.) by an electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The information processing device 1 may acquire the above program 81 from the storage medium 91. Note that the storage medium 91 may be a disk-type storage medium such as a CD, a DVD, etc., or may be a non-disk-type storage medium such as a semiconductor memory (for example, a flash memory). The drive 15 may be connected via an external interface. The external interface is configured to connect to an external device by wire or wirelessly.

[0043] Regarding the specific hardware configuration of the information processing device 1, depending on the embodiment, omission, replacement, and addition of components are possible as appropriate. For example, the control unit 11 may include a plurality of hardware processors. The hardware processors are a microprocessor, an FPGA (field-programmable gate array), a DSP (digital signal processor), GP It may be configured by a GPU (Graphics Processing Unit), an ASIC (application specific integrated circuit), etc. The storage unit 12 may be composed of a RAM and a ROM included in the control unit 11. It may also be composed of. At least one of the input device 13, the output device 14, and the drive 15 may be omitted. For example, the information processing apparatus 1 may be connected to another computer via an external interface, and input / output may be performed via the other computer. In this case, the output destination of the target image or the like may be the display of another computer, and the input device 13 and the output device 14 may be omitted. Note that the external interface may be, for example, a USB (Universal Serial Bus) port, a communication port, a dedicated port, or the like. The type and number of the external interfaces may be appropriately determined according to the embodiment. The information processing apparatus 1 may be, in addition to an information processing apparatus designed specifically for the provided service, a general-purpose server apparatus, a general-purpose PC (Personal Computer), a tablet PC, a mobile terminal including a smartphone, etc. and the like.

[0044] (Software configuration example) FIG. 8 schematically shows an example of the software configuration of the information processing apparatus 1 according to the present embodiment. The control unit 11 of the information processing apparatus 1 expands the program 81 stored in the storage unit 12 in the RAM, and executes the instructions included in the program 81 by the CPU. As a result, the information processing apparatus 1 operates as a computer including an acquisition unit 111, a display processing unit 112, an operation reception unit 113, a deformation processing unit 114, and an output processing unit 115 as software modules. That is, in the present embodiment, each software module of the information processing apparatus 1 is realized by the control unit 11 (CPU).

[0045] The acquisition unit 111 is configured to acquire two or more groups of target images for the object. Each of the two or more groups of target images corresponds to each of the two or more defined cross-section groups. The display processing unit 112 is configured to display, on the display 141, the corresponding target images included in the two or more acquired groups of target images, the template shape indicating the classification of the structure of the object, and the corresponding one or more control points included in the corresponding control point group, for the target cross-section among the one or more cross-sections included in the cross-section group defined for the target direction among the two or more directions. The operation reception unit 113 is configured to receive a movement operation on one or more control points in the target cross-section. The deformation processing unit 114 is configured to deform the template shape according to the arrangement of the one or more control points included in each control point group after the movement operation. The output processing unit 115 is configured to output the deformed template shape as an annotation for the structure of the object shown in the two or more groups of target images after the movement operation is completed.

[0046] Note that, in the present embodiment, an example in which each software module of the information processing apparatus 1 is realized by a general-purpose CPU is described. However, part or all of the above software modules may be realized by one or more dedicated processors or chip sets. Each of the above modules may be realized as a hardware module. Regarding the software configuration of the information processing apparatus 1, omission, replacement, and addition of modules may be appropriately performed.

[0047] §3 Operation Example FIG. 9 is a flowchart showing an example of the processing procedure of the information processing apparatus 1 according to the present embodiment. The following processing procedure is an example of an information processing method executed by a computer. However, the following processing procedure is merely an example, and each step may be changed as much as possible. Also, regarding the following processing procedure, omission, replacement, and addition of steps are possible as appropriate according to the embodiment.

[0048] In step S101, the control unit 11 operates as an acquisition unit 111 to acquire two or more target image groups for the object. In one example, the control unit 11 may acquire three target image groups, namely, a target image group corresponding to a horizontal plane group, a target image group corresponding to a sagittal plane group, and a target image group corresponding to a coronal plane group. The information source may not be particularly limited and may be appropriately determined according to the embodiment. In one example, the control unit 11 may acquire two or more target image groups by reading data held in the storage unit 12 or the storage medium 91. Further, the control unit 11 may acquire two or more target image groups from a device that generates a target image group or an external computer. The device that generates a target image group may be, for example, an MRI (Magnetic Resonance Imaging) device, a CT device, a micro-CT device, an ultrasonic diagnostic device, an optical coherence tomography device, etc. After acquiring two or more target image groups the control unit 11 proceeds to the next step S102.

[0049] In step S102, the control unit 11 operates as a display processing unit 112 to display, on the display 141, for the target cross-section, the corresponding target image, the corresponding partial shape of the template shape, and the corresponding one or more control points (control point set). The target cross-section to be displayed may be specified by the operator or automatically selected by the computer. Also, in step S102, the selection and display of the target cross-section may be repeatedly executed. By outputting each piece of information regarding the target cross-section, the control unit 11 may start accepting a movement operation on the control points on the target cross-section. After outputting each piece of information regarding the target cross-section, the control unit 11 proceeds to the next step S103.

[0050] In step S103, the control unit 11 determines whether to end the reception of the movement operation for the control points. The determination criterion may be set arbitrarily. In one example, the control unit 11 may determine whether to end the reception of the movement operation according to the presence or absence of a predetermined end operation (such as pressing an end button). If it is determined not to end the reception of the movement operation, the control unit 11 proceeds to the next step S104. If it is determined to end the reception of the movement operation, the control unit 11 proceeds to step S106.

[0051] In step S104, the control unit 11 operates as the operation reception unit 113 and receives a movement operation for one or more control points in the target cross-section. In step S105, the control unit 11 operates as the deformation processing unit 114 and deforms the template shape according to the arrangement of one or more control points included in each control point group after the movement operation. The control unit 11 may repeatedly execute the processes of step S104 and step S105 in the same target cross-section. When the deformation of the template shape is completed, the control unit 11 returns the process to step S102 and repeats the execution of the process from step S102.

[0052] Note that when returning to step S102, the control unit 11 may receive from the operator a change in the target direction and the target cross-section or only the target cross-section. The control unit 11 may execute each process after step S102 for the changed target cross-section. Also, at any timing, the control unit 11 may receive an end operation. In response to this, the control unit 11 may proceed to step S106.

[0053] In step S106, the control unit 11 operates as an output processing unit 115, and after the movement operation is completed, outputs the deformed template shape as an annotation for the structure of the object shown in two or more target image groups. The output form of the template shape may not be particularly limited and may be appropriately selected according to the embodiment. In one example, the control unit 11 may directly output the deformed template shape to the output device 14 (such as the display 141) or the output device of an external computer. Also, in another example, outputting the deformed template shape may include storing the deformed template shape in a predetermined storage area. The predetermined storage area may not be particularly limited and may be appropriately selected according to the embodiment. The predetermined storage area may be, for example, the RAM in the control unit 11, the storage unit 12, an external storage device, a storage medium, or a combination thereof. The storage medium may be, for example, a CD, a DVD, a semiconductor memory, etc., and the control unit 11 may store the template shape data in the storage medium via the drive 15. The external storage device may include the storage device of an external computer. The external storage device may be, for example, a data server such as a NAS. In this case, the control unit 11 may store the deformed template shape data in the data server via the network using an external interface (communication port). Also, the external storage device may be, for example, an external storage device. The external storage device may be appropriately connected to the information processing device 1. For example, the information processing device 1 may be connected to the external storage device via an external interface, and may store the deformed template shape data in the connected external storage device. The control unit 11 may store the generated deformed template shape in association with the two or more acquired target image groups. When the output of the template shape is completed, the control unit 11 ends the processing procedure of the information processing device 1 according to this operation example.

[0054] [Features] In this embodiment, by steps S104 and S105, as annotation work for the structure of the object, deformation work of the template shape by moving operation of each control point in each cross section in each direction is accepted. As exemplified in FIG. 5A and the like, this deformation work can be performed by an intuitive operation of moving the control points so as to match the structure of the object shown in the target image of the target cross section and deforming the template shape. Therefore, this deformation work is easy. Further, in this embodiment, since the control point groups are given independently for each direction, disappearance of control points from cross sections in each direction can be prevented. That is, it is possible to prevent losing the control points in any cross section while repeating the processes of steps S104 and S105. Therefore, according to this embodiment, annotation can be given by an easy method.

[0055] §4 Modification Example Although the embodiments of the present disclosure have been described in detail above, the foregoing description is merely illustrative of the present disclosure in every aspect. Needless to say, various improvements or modifications can be made without departing from the scope of the present disclosure. The processes and means described in the present disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0056] In each cross-section of the above-described embodiment, a series of processes for performing a movement operation on a control point and deforming a template shape may be used for purposes other than annotation work. In one example, the series of processes for deforming the above template shape may be used for deforming a 3D model. Accordingly, the object of the present disclosure may be to deform a 3D model in an easy manner. In this case, acquisition of two or more groups of target images and display of the target images may be omitted. Alternatively, acquisition of two or more groups of target images and display of the target images may be executed to show a sample of the 3D model after deformation. The template shape may indicate a division of the structure of the object, or may indicate any other information. In this modification example, the template shape corresponds to the 3D model. The template shape may be appropriately given according to the embodiment. The deformed template shape may be output for purposes other than annotation. According to this modification example, a 3D model can be deformed in an easy manner. This modification example may be used, for example, for the work of deforming the shape of clothing (such as sleeves and torso) to fit the target person.

Explanation of Signs

[0057] 1… Information processing apparatus, 211… Horizontal plane, 221… Sagittal plane, 231… Coronal plane, 31·32·33… Control point groups, 311·321·331… Control points, 5… Template shape, 611·621·631… Target images

Claims

1. A program for causing a computer to execute an information processing method, wherein each of two or more cross-section groups is defined for each of two or more directions, each of two or more control point groups is independently defined corresponding to each of the two or more cross-section groups, each of the cross-section groups includes one or more cross-sections in the defined direction, each of the control point groups includes one or more control points arranged for each of the cross-sections included in the corresponding cross-section group, and the information processing method includes: a step of acquiring two or more target image groups for an object, wherein each of the two or more target image groups corresponds to each of the two or more defined cross-section groups, a step of displaying, on a display, corresponding target images included in the acquired two or more target image groups, a template shape indicating a classification of the structure of the object, and corresponding one or more control points included in the corresponding control point group, for a target cross-section among the one or more cross-sections included in the cross-section group defined for a target direction among the two or more directions, a step of receiving a moving operation for the one or more control points in the target cross-section, a step of deforming the template shape according to the arrangement of the one or more control points included in each of the control point groups after the moving operation, and a step of outputting the deformed template shape as an annotation for the structure of the object shown in the two or more target image groups after the moving operation is completed. A program.

2. The object is a biological tissue, and each of the target image groups is any one of a magnetic resonance image group, a computed tomography image group, a micro-computed tomography image group, an ultrasonic image group, and an optical coherence tomography image group. The program according to claim 1.

3. The biological tissue is a musculoskeletal system. The program according to claim 2.

4. The two or more cross-section groups include a horizontal plane group, a sagittal plane group, and a coronal plane group. The program according to any one of claims 1 to 3.

5. An information processing method executed by a computer, wherein each of two or more cross-section groups is defined for each of two or more directions, each of two or more control point groups is independently defined corresponding to each of the two or more cross-section groups, each of the cross-section groups includes one or more cross-sections in the defined direction, each of the control point groups includes one or more control points arranged for each of the cross-sections included in the corresponding cross-section group, and the information processing method includes: ​ A step of acquiring two or more target image groups for an object, wherein each of the two or more target image groups corresponds to each of the two or more defined cross-section groups, For a target cross-section among the one or more cross-sections included in the cross-section group defined for a target direction among the two or more directions, a step of displaying on a display a corresponding target image included in the two or more acquired target image groups, a template shape indicating a classification of the structure of the object, and one or more corresponding control points included in the corresponding control point group, A step of receiving a movement operation on the one or more control points in the target cross-section, A step of deforming the template shape according to the arrangement of the one or more control points after the movement operation, and A step of outputting the deformed template shape as an annotation for the structure of the object shown in the two or more target image groups after the movement operation is completed, Including, An information processing method.

Images