Teacher data generation device, teacher data generation method, trained model manufacturing method, and program

By generating 3D and 2D annotations on three-dimensional and two-dimensional slice images, the complexity of lesion site annotation is reduced, enabling efficient creation of training data for machine learning models to identify lesion sites.

JP7595292B2Active Publication Date: 2024-12-06GLORY LTD +2
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2020145345
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-08-31
Publication Date
2024-12-06
Estimated Expiration
2040-08-31

AI Technical Summary

Technical Problem

The existing methods for annotating lesion sites in multiple two-dimensional slice images require complex and time-consuming operations, making it inefficient for training machine learning models.

Method used

A technique that generates teacher data by adding 3D annotations to three-dimensional surface data based on two-dimensional slice images, followed by converting these annotations to two-dimensional annotations, and using them to create training data with bounding boxes for known lesion sites.

Benefits of technology

Reduces the complexity of annotation operations and allows for the generation of a large number of training data efficiently, improving the accuracy and efficiency of machine learning models in identifying lesion sites.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007595292000001
    Figure 0007595292000001
  • Figure 0007595292000002
    Figure 0007595292000002
  • Figure 0007595292000003
    Figure 0007595292000003
Patent Text Reader

Abstract

To provide a technology to reduce the complexity in an annotation imparting operation.SOLUTION: An image processing device generates three-dimensional surface data 330 on the basis of a plurality of two-dimensional slice images 220, and receives an imparting operation of 3D annotation (annotation having three-dimensional position information) for the three-dimensional surface data 330. Also, the image processing device executes machine learning using data reflecting the annotation for the three-dimensional surface data 330 as teacher data. For example, the image processing device executes first processing for imparting 3D annotation 53 for the three-dimensional surface data according to an operation of an operating user, and executes second processing for automatically imparting 2D annotation in the two-dimensional slice images 220 on the basis of the 3D annotation 53. The data based on the two-dimensional slice images 220 to which the 2D annotation is imparted is used as teacher data, and machine learning is executed.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to processing techniques and related techniques for multiple two-dimensional slice images. [Background technology]

[0002] There is a technique for acquiring a plurality of two-dimensional slice images (also called cross-sectional images) of a subject using an MRI (Magnetic Resonance Imaging) device or a CT (Computed Tomography) device, and identifying a lesion using the plurality of two-dimensional slice images.

[0003] For example, Patent Document 1 describes a technique for determining the presence or absence of a lesion from a plurality of two-dimensional slice images by machine learning using a neural network. In such a technique, a large number of two-dimensional slice images including known lesion sites are used as teacher data for training a machine learning learner. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2019-504659 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above-mentioned technique, an operation of adding annotation marks to lesion sites (fracture sites, etc.) is performed in each of a large number (eg, several hundred) of two-dimensional slice image data for each subject (each individual).

[0006] Such an operation is complicated and takes a huge amount of time.

[0007] Therefore, an object of the present invention is to provide a technique capable of reducing the complexity of annotation assignment operations. [Means for solving the problem]

[0008] In order to solve the above problems, The teacher data generation device of the present invention is a teacher data generation device that generates teacher data for machine learning a learning model that identifies lesion sites in a target image when a target image is input, and is equipped with a control unit that generates 3D surface data based on a plurality of 2D slice images, and a reception unit that receives an operation to add a 3D annotation, which is an annotation for a lesion site in the 3D surface data and has three-dimensional position information, wherein the control unit executes a first process to add the 3D annotation to the 3D surface data in response to the addition operation, executes a second process to add a 2D annotation, which is an annotation having two-dimensional position information, in at least one 2D slice image of the plurality of 2D slice images based on the 3D annotation, and generates teacher data having a bounding box indicating a known lesion site in the at least one 2D slice image based on the 2D annotation.

[0009] The receiving unit may receive an operation for adding a 3D annotation over a range spanning two or more of the plurality of 2D slice images.

[0010] The receiving unit may receive an operation of specifying a two-dimensional position on a projection surface onto which the three-dimensional surface data is projected as an operation of specifying a position of the 3D annotation, and the control unit may convert the two-dimensional position on the projection surface into a three-dimensional position on the three-dimensional surface data, and generate three-dimensional position information of the 3D annotation.

[0013] The 2D annotation may be assigned to a corresponding two-dimensional position, which is a two-dimensional position corresponding to the three-dimensional position of the 3D annotation, in at least one of the two-dimensional slice images corresponding to the three-dimensional position of the 3D annotation, and may also be assigned to a two-dimensional position in each of the two-dimensional slice images that is determined to be the corresponding two-dimensional position in any of a predetermined number of two-dimensional slice images in the vicinity of each of the two-dimensional slice images.

[0014] The 2D annotation may include a 2D basic annotation that is assigned to a two-dimensional position in the at least one two-dimensional slice image that corresponds to the three-dimensional position of the 3D annotation, and a 2D complementary annotation that complements the 2D basic annotation, the 2D complementary annotation being assigned to each two-dimensional slice image at a two-dimensional position where the 2D basic annotation was assigned in any of a predetermined number of slice images in the vicinity of each two-dimensional slice image.

[0015] The control unit controls the 3D annotation as the 2D annotation. 3 a 2D extended annotation is assigned not only to a corresponding two-dimensional position, which is a two-dimensional position corresponding to the three-dimensional position, but also to other positions, and the 2D extended annotation may include a 2D basic annotation assigned to a two-dimensional position corresponding to the three-dimensional position in at least one of the two-dimensional slice images corresponding to the three-dimensional position of the 3D annotation, and a 2D complementary annotation assigned to a two-dimensional position in each of the two-dimensional slice images to which the 2D basic annotation is assigned in any of a predetermined number of two-dimensional slice images in the vicinity of each of the two-dimensional slice images.

[0016] The 2D annotation may be assigned to all two-dimensional positions within each two-dimensional slice image that are determined to be two-dimensional positions that correspond to the three-dimensional position of the 3D annotation in either that two-dimensional slice image or a predetermined number of nearby two-dimensional slice images.

[0017] The control unit uses partial images in at least some of the two-dimensional slice images to which two or more 2D annotations are added, the partial images being within bounding boxes corresponding to the two or more 2D annotations, as the teacher data. It may be generated.

[0018] The control unit uses as the training data a partial image within at least a part of the two-dimensional slice images among the plurality of two-dimensional slice images, the partial image being within a bounding box corresponding to both the 2D basic annotation and the 2D supplemental annotation. It may be generated.

[0020] The three-dimensional surface data may be data representing a three-dimensional surface shape of a bone.

[0023] In order to solve the above problem, the teacher data generation method of the present invention is a teacher data generation method for generating teacher data for machine learning a learning model that identifies a lesion site in a target image when a target image is input, and is characterized by comprising: a) a step in which a computer receives an operation to add a 3D annotation, which is an annotation for a lesion site in 3D surface data generated based on a plurality of 2D slice images and has three-dimensional position information; b) a step in which the computer adds the 3D annotation to the 3D surface data in response to the addition operation; c) a step in which the computer adds a 2D annotation, which is an annotation having two-dimensional position information, in at least one 2D slice image of the plurality of 2D slice images based on the 3D annotation; and d) a step in which the computer generates the teacher data having a bounding box indicating a known lesion site in the at least one 2D slice image based on the 2D annotation.

[0024] In step c), the 2D annotation is assigned to a corresponding 2D position in at least one slice image, which is a 2D position that corresponds to the 3D position of the 3D annotation, and may also be assigned to each 2D slice image at a 2D position determined to be the corresponding 2D position in any of a predetermined number of 2D slice images in the vicinity of each 2D slice image.

[0025] In order to solve the above problem, the present invention provides a method for manufacturing a trained model, which produces a trained model by machine learning a training model that identifies a lesion site in a target image when a target image is input, and which includes the steps of: a) a step in which a computer receives an operation for adding a 3D annotation, the 3D annotation being an annotation for a lesion site in 3D surface data generated based on a plurality of 2D slice images and having 3D position information; b) a step in which the computer adds the 3D annotation to the 3D surface data in response to the addition operation; c) a step in which the computer adds a 2D annotation, the 2D annotation being an annotation having 2D position information, in at least one 2D slice image of the plurality of 2D slice images based on the 3D annotation; d) a step in which the computer generates training data based on the 2D annotation, the training data having a bounding box indicating a known lesion site in the at least one 2D slice image; and e) a step in which the computer uses the training data related to the at least one 2D slice image to machine learn the training model, thereby generating the trained model.

[0026] In the step c), the 2D annotation is assigned to a corresponding two-dimensional position in the at least one slice image, which is a two-dimensional position corresponding to the three-dimensional position of the 3D annotation, and is also assigned to a two-dimensional position in each of the two-dimensional slice images that is determined to be the corresponding two-dimensional position in any of a predetermined number of two-dimensional slice images in the vicinity of each of the two-dimensional slice images, and the step e) comprises the steps of: e-1) performing machine learning using, as the training data, partial images in at least some of the two-dimensional slice images to which two or more 2D annotations are assigned, the partial images being within bounding boxes corresponding to the two or more 2D annotations; Run The method may further include a step of:

[0027] In order to solve the above problem, the program of the present invention is a program for causing a computer to execute a process of generating teacher data for machine learning a learning model that identifies a lesion site in a target image when the target image is input, and is characterized in that the program causes the computer to execute the following steps: a) receiving an operation to add a 3D annotation, which is an annotation for a lesion site in 3D surface data generated based on a plurality of 2D slice images and has three-dimensional position information; b) adding the 3D annotation to the 3D surface data in response to the addition operation; c) adding a 2D annotation, which is an annotation having two-dimensional position information, in at least one 2D slice image of the plurality of 2D slice images based on the 3D annotation; and d) generating the teacher data having a bounding box indicating a known lesion site in the at least one 2D slice image based on the 2D annotation.

[0028] In order to solve the above problem, the program of the present invention is a program for causing a computer to execute a process of machine learning a learning model that identifies a lesion site in a target image when the target image is input, and is characterized in that the program causes the computer to execute the following steps: a) receiving an operation to add a 3D annotation, which is an annotation for a lesion site in 3D surface data generated based on a plurality of 2D slice images and has three-dimensional position information; b) adding the 3D annotation to the 3D surface data in response to the addition operation; c) adding a 2D annotation, which is an annotation having two-dimensional position information, in at least one 2D slice image of the plurality of 2D slice images based on the 3D annotation; d) generating teacher data having a bounding box indicating a known lesion site in the at least one 2D slice image based on the 2D annotation; and e) machine learning the learning model using the teacher data related to the at least one 2D slice image.

[0029] In any of the above programs, step a) may comprise a step of accepting a designation operation of a two-dimensional position on a projection surface onto which the three-dimensional surface data is projected as an operation of designating a position of the 3D annotation, and step b) may comprise a step of converting the two-dimensional position on the projection surface into a three-dimensional position on the three-dimensional surface data, and generating three-dimensional position information of the 3D annotation. Effect of the Invention

[0030] According to the present invention, it is possible to reduce the complexity of the annotation assignment operation. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a block diagram showing an image processing system. [Diagram 2] FIG. 1 is a conceptual diagram showing the preparatory processing and data of teacher data. [Diagram 3] FIG. 13 is a conceptual diagram showing the processing in the learning stage. [Figure 4] FIG. 1 is a conceptual diagram showing the processing at the inference stage. [Diagram 5] 13 is a flowchart showing the process of the controller in a preparation stage. [Figure 6] 10 is a flowchart showing the process of the controller in a learning stage. [Figure 7] 13 is a flowchart showing the process of the controller in the inference stage. [Figure 8] FIG. 2 is a schematic diagram showing a plurality of two-dimensional slice images. [Figure 9] FIG. 2 is a schematic diagram showing three-dimensional surface data, etc. [Figure 10] FIG. 1 is a schematic diagram showing three-dimensional surface data to which 3D annotations have been added. [Figure 11]FIG. 1 is a conceptual diagram showing 2D annotations etc. added based on 3D annotations. [Figure 12] FIG. 13 is a partial enlarged view of a three-dimensional model with a complete fracture. [Figure 13] FIG. 13 is a partially enlarged view of a three-dimensional model with an incomplete fracture. [Figure 14] FIG. 11 is a schematic diagram for explaining a process of adding a 3D annotation. [Figure 15] 1A to 1C are diagrams showing a projected image projected onto a projection surface, etc.; [Figure 16] 11A to 11C are diagrams showing a process of adding 3D annotations and a process of adding 2D annotations, etc. [Figure 17] FIG. 13 is a diagram showing three-dimensional surface data to which another 3D annotation has been added. [Figure 18] FIG. 13 is a diagram showing three-dimensional surface data to which another 3D annotation has been added. [Figure 19] 19 is a diagram showing 2D annotations etc. in another cross section of the three-dimensional surface data of FIG. 18. [Figure 20] FIG. 13 is a diagram showing 2D annotations and the like in yet another cross section. [Figure 21] FIG. 13 is a diagram showing 2D annotations and the like in yet another cross section. [Figure 22] 10 is a flowchart showing a part of a process in a second embodiment. [Diagram 23] FIG. 2 shows a series of two-dimensional slice images. [Figure 24] FIG. 1 shows a series of 2D slice images with 2D basic annotations. [Diagram 25] FIG. 1 shows a series of 2D slice images with 2D extended annotations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0033] <1. First embodiment> <1-1. System Overview> 1 is a block diagram showing an image processing system 10. The image processing system 10 is a system that processes a plurality of two-dimensional slice images 220 obtained by slicing a subject (such as a human subject) along a cross section perpendicular to a reference axis.

[0034] The image processing system 10 includes a slice image generating device 20 and an image processing device 30.

[0035] The slice image generating device 20 is composed of an MRI (Magnetic Resonance Imaging) device or a CT (Computed Tomography) device. The slice image generating device 20 generates and acquires a plurality of two-dimensional slice images (also simply referred to as slice images) 220 of a subject (such as a human subject). The plurality of slice images 220 are images obtained by slicing the subject in a cross section perpendicular to a reference axis at a plurality of different positions on the reference axis (for example, at different positions with a pitch of 0.6 mm to 1 mm (millimeter)). The plurality of slice images 220 are acquired over a predetermined range (for example, 300 mm) in the reference axis direction, and are composed of several hundred to several thousand images (for example, 500 images).

[0036] The plurality of slice images 220 (also referred to as one slice image group 210) may be obtained for one specimen with respect to different reference axes (for example, nine different reference axes (nine directions)), forming a plurality of slice image groups. The number of slice image groups thus obtained is very large, consisting of several thousand or more images (for example, 4500 images). The plurality of slice image groups may be obtained, for example, by imaging slice image groups in each direction (for example, each of nine directions) with respect to the different reference axes. However, without being limited thereto, the plurality of slice image groups may be obtained by imaging one slice image group in one direction and performing image conversion processing on the one slice image group to generate slice image groups in other directions (for example, eight slice image groups with the other eight directions as the reference axis directions).

[0037] Furthermore, the plurality of slice images (one slice image group) or the plurality of slice image groups are acquired for a plurality of specimens, respectively, to configure a plurality of slice image groups for the plurality of specimens. The number of slice image groups for the plurality of specimens is enormous.

[0038] In the slice image generating device 20, for example, a slice image group 210 relating to the lumbar region including the pelvis is captured and acquired.

[0039] The image processing device 30 executes machine learning based on a plurality of two-dimensional slice images 220 (specifically, a group of a plurality of slice images related to a plurality of specimens) acquired by the slice image generating device 20. Specifically, the learning parameters of a learning model 410 (learning device) are adjusted using a predetermined machine learning method, and a trained model 410 (also referred to as 420) is generated (see FIG. 3). For example, a neural network model consisting of a plurality of layers is used as the learning model 410. Then, weighting coefficients (learning parameters) between layers (input layer, (one or more) intermediate layers, output layer) in the neural network model are adjusted by a predetermined machine learning method (deep learning, etc.). In this way, the image processing device 30 executes the processing of the learning stage in the machine learning. Note that FIG. 3 is a conceptual diagram showing the processing of the learning stage.

[0040] Specifically, the image processing device 30 generates a learning model (trained model 420) for executing an inference process for identifying a lesion site in a slice image based on teacher data (a group of a plurality of known slice images) having a lesion site (see Figs. 3 and 6). Here, a fracture site (more specifically, a fracture site in a pelvis) is exemplified as the lesion site. Note that Fig. 6 is a flowchart showing the process of the controller 31 (see Fig. 1) in the learning stage.

[0041] The image processing device 30 also executes processing related to the preparatory stage of teacher data (preparatory processing for generating teacher data to be used in the learning stage). Specifically, processing for adding annotations to three-dimensional surface data (three-dimensional surface data of bones) is executed (see Figs. 2 and 5). Fig. 2 is a conceptual diagram showing processing and data in the preparatory stage of teacher data, and Fig. 5 is a flowchart showing processing of the controller 31 (Fig. 1) in the preparatory stage.

[0042] Furthermore, the image processing device 30 executes an inference process for each of a plurality of (unknown) slice images 220 (also referred to as 270) acquired for a certain subject, using the learning model 410 (trained model 420) whose parameters have been adjusted (see Figs. 4 and 7). Specifically, the image processing device 30 executes an inference process for identifying a lesion site (fracture site) in each slice image 270, using the above-mentioned trained model 420. In the inference process, for example, the presence or absence of a fracture site and the position of the fracture site are identified (estimated). Note that Fig. 4 is a conceptual diagram showing the process at the inference stage in machine learning, and Fig. 7 is a flowchart showing the process of the controller 31 (Fig. 1) at the inference stage.

[0043] 1, the image processing device 30 includes a controller 31 (also referred to as a control unit), a storage unit 32, and an operation unit 35. The slice image generating device 20 and the image processing device 30 are connected by wire (or wirelessly) and can communicate with each other. The image processing device 30 receives information (slice images, etc.) acquired by the slice image generating device 20 from the slice image generating device 20 via a predetermined connection cable.

[0044] The controller 31 is a control device that is built into the image processing device 30 and controls the operation of the image processing device 30 .

[0045] The controller 31 is configured as a computer system including a single or multiple CPUs (Central Processing Units) (also referred to as microprocessors or hardware processors, etc.). The controller 31 realizes various processes by executing, in the CPU, a predetermined software program (hereinafter also simply referred to as a program) stored in a storage unit (a non-volatile storage unit such as a ROM and / or a hard disk) 32. The program (specifically, a group of program modules) may be recorded in a portable recording medium such as a USB memory, read from the recording medium, and installed in the image processing device 30. Alternatively, the program may be downloaded via a communication network or the like and installed in the image processing device 30.

[0046] The controller 31 executes processes related to the preparation stage of teacher data in machine learning. Specifically, the controller 31 executes processes of acquiring a plurality of slice images 220 from the slice image generating device 20, generating three-dimensional surface data 330, adding annotations to the three-dimensional surface data 330, and generating teacher data based on the annotations.

[0047] The controller 31 also executes processing related to the learning stage in machine learning. Specifically, processing is executed to optimize learning parameters related to the learning device (learning model 410) based on the generated teacher data, and a trained model 420 is generated. More specifically, machine learning is performed using a slice image 220 having a known fracture site as teacher data, and the learning parameters are optimized. As a result, the trained model 420 is generated.

[0048] Furthermore, the controller 31 executes processing related to the inference stage in machine learning. Specifically, using a learning model (trained model 420) with adjusted parameters, an inference process is executed for each of a plurality of (unknown) slice images 270 acquired for a certain subject. In particular, an inference process is executed to identify a lesion site (fracture site) in each slice image 270. The controller 31 also executes processing to output an inference result (such as display processing of an image including a fracture site).

[0049] The storage unit 32 is configured with a storage device such as a hard disk drive (HDD) and / or a solid state drive (SSD), etc. The storage unit 32 stores a plurality of 2D slice images 220, 260, 270, 280, 3D surface data 330, and a learning model 410 (including learning parameters related to the learning model) (and thus a learned model 420), etc.

[0050] The operation unit 35 includes an operation input unit 35a that receives an operation input to the image processing device 30, and a display unit 35b that displays and outputs various information. For example, the operation input unit 35a (also referred to as a reception unit) receives an operation of adding an annotation (3D annotation 53 (see FIG. 10, etc.)) to the three-dimensional surface data. The display unit 35b displays a slice image 280 related to the result of the inference process using the trained model 420. A mouse and a keyboard are used as the operation input unit 35a, and a display (such as a liquid crystal display) is used as the display unit 35b. A touch panel that functions as both a part of the operation input unit 35a and a part of the display unit 35b may be provided.

[0051] The image processing device 30 is also called a medical image processing device, and the image processing system 10 is also called a medical image processing system.

[0052] Below, we will explain each process in turn: the stage of preparing the training data, the stage of learning using the training data, and the stage of inference.

[0053] <1-2. Preparation of training data> <Processing Overview> First, an overview of the preparatory process for the teacher data will be given.

[0054] In the machine learning according to the present embodiment, a plurality of slice images 220 (slice image group 210) of a subject (individual) having a fracture condition (fracture disease) are used as training data in training with training data. More specifically, a plurality of two-dimensional slice images 220 (a plurality of slice image groups 210) of a plurality of subjects having a fracture condition are used. In particular, a two-dimensional slice image 220 including a fracture site is used. In detail, an area including a known fracture site (also referred to as a fracture site area) is identified in the two-dimensional slice image 220, and then the two-dimensional slice image 220 including the fracture site area is used.

[0055] One method for identifying a fracture site region in a two-dimensional slice image is for an operator to directly add annotation marks to the fracture site (lesion site) in each two-dimensional slice image. This method requires an operator to add annotation marks to the fracture site in each of a large number (e.g., several hundred) of two-dimensional slice image data.

[0056] However, the operation of an operating user adding annotations to each lesion site (such as a fracture site) of a large number (eg, several hundred) of two-dimensional slice images for each two-dimensional slice image is extremely cumbersome and requires a huge amount of time.

[0057] Therefore, in this embodiment, a new method is used to add annotations to the two-dimensional slice image 220 (also simply referred to as a slice image), which can reduce the complexity of the annotation addition operation.

[0058] The new method is a method of adding annotations by using three-dimensional surface data 330 (see FIGS. 2 and 9) based on a plurality of two-dimensional slice images 220 (see FIG. 8, etc.). More specifically, the image processing device 30 accepts an adding operation on the three-dimensional surface data 330 by an operating user. Then, the image processing device 30 adds a 3D (three-dimensional) annotation 53 (annotation having three-dimensional position information) to the surface of the three-dimensional surface data 330 (a three-dimensional model of a human body structure (bones)) based on the adding operation (see FIGS. 2 and 10, etc.). Furthermore, the image processing device 30 adds a 2D (two-dimensional) annotation 52 (annotation having information on a two-dimensional position in each slice image) to each of the plurality of two-dimensional slice images 220 based on the 3D annotation 53 (see FIGS. 2 and 11, etc.).

[0059] FIG. 8 is a schematic diagram showing a plurality of two-dimensional slice images. FIG. 9 is a schematic diagram showing three-dimensional surface data 330 and the like generated based on a plurality of two-dimensional slice images 220 (see FIG. 8 and the like). In FIG. 9, the three-dimensional surface data 330 is shown in a simplified form, but in reality, three-dimensional surface data 330 is generated that is precise enough to distinguish the state of the bone surface (to the extent that the fractured part can be visually recognized). FIG. 10 is a conceptual diagram showing three-dimensional surface data 330 (also referred to as 340) and the like to which a 3D annotation 53 is added, and FIG. 11 is a conceptual diagram showing 2D annotations 52 and the like added based on the 3D annotations 53. In FIG. 11, a two-dimensional slice image 220 (also referred to as 260) to which a 2D annotation 52 is added is shown, as well as a bounding box 55 (described later) surrounding the 2D annotation 52 (see the upper right part of FIG. 16 and the like). These figures (particularly Figures 10 and 11) show a fracture site on the left side of the subject's body. The fracture site is shown on the right side of Figure 11 and other figures depicting two-dimensional slice images (images in which the spine is positioned at the bottom (images viewed from the foot side, not the head side)), and is also shown on the right side of Figure 10 and other figures depicting three-dimensional surface data 330 (a three-dimensional model of the subject viewed from the front (front side)).

[0060] <Processing details> Such a technique will be described in further detail below with reference to FIG.

[0061] Fig. 5 is a flowchart showing the process (more specifically, the process in the preparation stage) of the controller 31. Note that Fig. 5 is also a diagram showing a method of adding annotations and a diagram showing (a part of) a method of generating a trained model.

[0062] First, in step S11 (FIG. 5), the controller 31 acquires a series of slice images 220 (see the top row of FIG. 2 and FIG. 8) generated (captured) by the slice image generating device 20. In detail, a series of slice images 220 relating to a certain subject (examinee) is acquired.

[0063] Next, in step S12, as shown in Fig. 9, the controller 31 generates three-dimensional surface data 330 (three-dimensional model) (see also the second row from the top in Fig. 2) based on the plurality of slice images 220. This three-dimensional surface data 330 is data that represents the surface of a three-dimensional model of a part of the subject (here, a three-dimensional model that represents the subject's pelvic bone). The three-dimensional surface data 330 is configured, for example, as polygon mesh data that represents a surface by gathering a large number of adjacent polygons (e.g., triangles). Here, polygon mesh data that represents the three-dimensional surface shape of the subject's bone (pelvis) is generated as the three-dimensional surface data 330.

[0064] The three-dimensional surface data 330 is formed as follows: For example, a bone surface region is extracted by image processing on each of the slice images 220, and the surface regions of the slice images 220 are stacked in the reference axis direction (normal direction) of the slice images. Then, the three-dimensional surface data 330 is formed by, for example, connecting the surface regions adjacent to each other in the stacking direction.

[0065] In the next step S13, the controller 31 accepts an operation for adding an annotation (hereinafter also referred to as a "3D annotation") 53 that is an annotation to the three-dimensional surface data 330 and has three-dimensional position information. Furthermore, the controller 31 (automatically) adds the 3D annotation 53 to the three-dimensional surface data 330 (specifically, to the surface of a three-dimensional model (solid model) related to the three-dimensional surface data 330) in response to the operation.

[0066] Fig. 14 is a schematic diagram for explaining the process of adding such a 3D annotation 53. Note that the three-dimensional surface data 330 is actually a three-dimensional surface model of a pelvis having a very complicated structure, but in Fig. 14, for the sake of simplicity, the three-dimensional surface data 330 is shown as a cylindrical three-dimensional surface model.

[0067] The controller 31 projects and displays the three-dimensional surface data 330 (in other words, a three-dimensional surface model of the bone) on a projection plane 70 that is virtually arranged between the position of the three-dimensional surface data 330 (the position in the virtual space) and the viewpoint position of the operating user. FIG. 15 is a diagram showing a state in which such a projection plane 70 is displayed on the display unit 35b. The three-dimensional surface data 330 (three-dimensional surface model of the bone) is rotated, moved (translated), and scaled (enlarged and / or reduced) as appropriate in response to a user operation. In other words, the user can visually recognize the three-dimensional surface data 330 (three-dimensional surface model of the bone) projected onto the projection plane 70 at a desired size from a desired viewpoint.

[0068] Display unit 35b displays a projection image 332 (two-dimensional image) obtained by projecting three-dimensional surface data 330 onto projection surface 70. In Fig. 15, a rectangular projection image 332 is displayed by viewing a cylindrical solid surface model (three-dimensional surface data 330) with its central axis in the vertical direction from the front.

[0069] In such a projection image 332, the operating user draws an annotation line 51 on a portion that is determined to be a fracture site from a medical perspective. The annotation line 51 is drawn in a line shape (straight or curved) by the operating user using a mouse or the like (operation input unit 35a), for example.

[0070] Such a drawing operation of the annotation line 51 (including an operation of specifying a two-dimensional position on the projection surface 70 onto which the three-dimensional surface data 330 is projected) is accepted as an operation of specifying the position of the 3D annotation 53. Then, when the annotation line 51 is drawn on the projection surface 70, the controller 31 converts the two-dimensional position of the annotation line 51 on the projection surface 70 into a three-dimensional position on the three-dimensional surface data 330, and generates three-dimensional position information of the 3D annotation 53.

[0071] 14 (particularly see the dashed arrow), each point on the annotation line 51 is moved toward the three-dimensional surface data 330 in the user's line of sight. Then, the position where each point moving in the line of sight hits (reaches) the surface of the three-dimensional surface data 330 (the surface of the three-dimensional model of the bone) is calculated as the three-dimensional position of the 3D annotation 53. FIG. 14 shows a situation where the linear annotation line 51 on the projection plane 70 is associated with each position on the surface (curved surface) of the three-dimensional surface data 330. A portion represented by a curve associated with the surface of the three-dimensional surface data 330 is added as the 3D annotation 53.

[0072] The annotation line 51 is actually drawn on the surface of a three-dimensional model of the pelvis (three-dimensional surface data 330), for example, as shown in Fig. 10. The upper part of Fig. 10 shows the three-dimensional surface data 330 (three-dimensional model of the pelvis) generated in step S12, and the lower part of Fig. 10 shows the three-dimensional surface data 330 on which the annotation line 51 is drawn.

[0073] More specifically, an annotation line 51 as shown in FIG. 12 or 13 is drawn. The annotation line 51 is drawn as a line having a predetermined width (the width of several polygons (e.g., 3 to 4 polygons)) in the three-dimensional surface data 330 configured as polygon mesh data (see FIG. 12 and FIG. 13). In other words, it is drawn as a long and narrow planar region having a predetermined width. The predetermined width may be changed as appropriate in response to a designation operation (changing operation) by the operating user, etc.

[0074] 12 and 13 are partial enlarged views of the solid model (three-dimensional surface data 330). The fracture state is shown typically in the upper part of Fig. 12 and the upper part of Fig. 13. The three-dimensional surface data 330 is shown as polygon mesh data in the lower part of Fig. 12 and the lower part of Fig. 13, and the fracture site on the polygon mesh data is shown.

[0075] Here, there are "complete fractures" and "incomplete fractures". A "complete fracture" (see Figure 12) is a fracture in which fragments have occurred (such as a fracture in which the bone is completely ruptured). On the other hand, an "incomplete fracture" (see Figure 13) is a fracture in which no fragments have occurred (such as a fracture in which the bone is not completely ruptured, such as a state in which a so-called "crack" has occurred). Note that Figure 12 is a partial enlarged view of a three-dimensional model with a complete fracture, and Figure 13 is a partial enlarged view of a three-dimensional model with an incomplete fracture.

[0076] As shown in Fig. 12, in the case of a complete fracture in which a fragment is present, annotation lines 51 are drawn on both sides of the fragment (the space between both interfaces of the fracture site) (the black part in the center of the lower part of Fig. 12). On the other hand, as shown in Fig. 13, in the case of an incomplete fracture in which no fragment is present, annotation lines 51 are drawn along the part determined to be the fracture site (the line determined by a doctor or the like to be the fracture site) (see the dashed line in Fig. 13).

[0077] Simultaneously with drawing the annotation line 51, the three-dimensional position of the annotation line 51 is calculated, and a 3D annotation 53 is generated in accordance with the operation of calculating the three-dimensional position.

[0078] More specifically, when an operating user draws an annotation line 51 having a predetermined width on the projection image 332, a polygon existing at a position corresponding to the annotation line 51 of the predetermined width is specified in the three-dimensional surface data 330. Then, the annotation line 51 formed by a set of the polygons is drawn on the three-dimensional surface data 330 (on the surface of the three-dimensional surface data 330), and at the same time, the three-dimensional position of each of the polygons is specified as the three-dimensional position of the 3D annotation 53. Note that the annotation line 51 on the three-dimensional surface data 330 is also expressed as the 3D annotation 53. In this way, the 3D annotation 53 is generated in response to an operation of adding the 3D annotation 53 indicating the fracture site (an operation of drawing the annotation line 51 in the projection image 332).

[0079] Moreover, the annotation line 51 is drawn over a relatively long range (e.g., a range that goes halfway around or around the bone) on the surface of the bone to indicate the fractured portion (see Figs. 10, 16, and 17). As a result, an operation to add a 3D annotation 53 is accepted over a range that spans two or more of the multiple two-dimensional slice images 220 (a predetermined range in the normal direction of the slice cross section (e.g., a range that corresponds to a range from a minimum value Zmin to a maximum value Zmax in the Z-axis direction (the normal direction) (described later))). Then, in response to the adding operation, the 3D annotation 53 is added over a range that spans the two or more two-dimensional slice images.

[0080] FIG. 16 is a conceptual diagram for explaining the process of adding the 3D annotation 53 and the process of adding the 2D annotation 52 based on the 3D annotation 53. FIG. 16 is a diagram similar to FIG. 14. However, in FIG. 16, a curve different from the annotation line 51 in FIG. 14 is drawn as the annotation line 51. Similarly, FIG. 17 is a diagram similar to FIG. 14 and FIG. 16. However, in FIG. 17, a curve different from the annotation line 51 in FIG. 14 and FIG. 16 is drawn as the annotation line 51. Also, in each diagram after FIG. 16 (except FIG. 22), an image of a cross section (horizontal cross section) of a solid model (three-dimensional surface data 330) viewed from above is appropriately shown as the two-dimensional slice image 220.

[0081] In step S13, as described above, the 3D annotation 53 is generated based on the annotation line 51 drawn on the projection image 332. For example, in Fig. 16, the 3D annotation 53 (see the middle part of Fig. 16) corresponding to the annotation line 51 (see the bottom part of Fig. 16) drawn on the projection image 332 is added to the three-dimensional surface data 330.

[0082] In the next step S14, the controller 31 assigns the 2D annotation 52 based on the 3D annotation 53 to the two-dimensional slice image 220.

[0083] 16, a substantially semicircular 3D annotation 53 is added to the surface of a cylindrical solid model (three-dimensional surface data 330). Here, it is assumed that a fracture (including a crack, etc.) occurs near the surface where a plane (approximate plane) roughly including the substantially semicircular 3D annotation 53 intersects with a cylindrical space (space corresponding to a bone) whose surface is indicated by the three-dimensional surface data 330. In detail, the plane (approximate plane) including the substantially semicircular 3D annotation 53 intersects obliquely with the side surface of the cylindrical space (space corresponding to a bone). It is assumed that a fracture occurs near this intersection.

[0084] In step S14, the controller 31 automatically assigns the 2D annotation 52 to a two-dimensional position (also referred to as a "corresponding two-dimensional position") corresponding to the three-dimensional position of the 3D annotation 53 in at least one two-dimensional slice image corresponding to the three-dimensional position of the 3D annotation 53. Here, the 2D annotation 52 is an annotation having two-dimensional position information in each two-dimensional slice image.

[0085] In detail, first, the controller 31 finds at least one two-dimensional slice image 220 (preferably two or more two-dimensional slice images 220) that has an intersection line (or intersection point) with the 3D annotation 53, which is a long and narrow planar region, in its plane among the multiple two-dimensional slice images 220. For example, when the Z axis is arranged along the reference axis direction (normal direction of the slice cross section) of the slice image group 210, the existence range (maximum value Zmax and minimum value Zmin) of the 3D annotation 53 in the Z axis direction is found. Then, at least one two-dimensional slice image 220 from the slice image 220 having the minimum value Zmin to the slice image 220 having the maximum value Zmax is identified.

[0086] Furthermore, the controller 31 automatically assigns a 2D annotation to a two-dimensional position (corresponding two-dimensional position) corresponding to the three-dimensional position of the 3D annotation 53 in the at least one two-dimensional slice image 220. For example, when the Z axis is arranged along the reference axis direction (normal direction of the slice cross section) of the slice image group 210, the controller 31 obtains the two-dimensional position (two-dimensional position (XY position) of each intersection) of the intersection line (a set of intersection points) between each two-dimensional slice image 220 and the 3D annotation 53, which is an elongated planar region. Then, the controller 31 assigns the 2D annotation 52 to the two-dimensional position corresponding to the intersection line in each two-dimensional slice image 220. In response to this, the visualized 2D annotation 52 (specifically, a mark indicating the 2D annotation 52) is drawn at the two-dimensional position.

[0087] For example, as shown in FIG. 16, when the intersection lines between the two-dimensional slice image 220 and the 3D annotation 53 are obtained at two points in a certain two-dimensional slice image 220, the 2D annotation 52 is added to the two points (see the top of FIG. 16). More specifically, the 2D annotation 52 is added near the bottom end and near the right end of the circular region in the top of FIG. 16. The circular region (region with sandy hatching) in the top of FIG. 16 represents a cut surface (cut surface by the two-dimensional slice image 220 (plane)) of a three-dimensional model of a bone expressed (schematically) in a cylindrical shape, that is, a cross section of the bone. The curved line inside the circular region represents a fracture line (a portion where a crack or the like of the bone appears as a linear region in the two-dimensional slice image (cross-sectional image)). The region surrounded by an elliptical dashed line in the middle of FIG. 16 corresponds to the circular region in the top of FIG. 16.

[0088] Then, the controller 31 draws a bounding box 55 so as to include the two 2D annotations 52, and determines a rectangular area enclosed by the bounding box 55 as training data indicating the fracture area.

[0089] In this way, in at least one two-dimensional slice image 220, the 2D annotation 52 is added to a two-dimensional position corresponding to the intersection line with the 3D annotation 53. In addition, a rectangular area (partial image) that surrounds the 2D annotation 52 with a banding box is determined as training data indicating a fracture area.

[0090] As described later, depending on the way of drawing the annotation line 51, the intersection line with the 3D annotation 53 may appear at only one location in a certain two-dimensional slice image 220 (slice cross section). In this case, for example, a bounding box 55 may be formed so as to surround only the one 2D annotation 52. Here, it is assumed that only a single fracture site exists for one subject. When multiple fracture sites exist for one subject, each fracture site may be assigned (labeled) with an identifier (identification number / identification symbol, etc.) for identifying each fracture site, so that the multiple fracture sites can be distinguished from one another. Then, the above-mentioned processes of steps S13 and S14 may be performed on each of the multiple fracture sites distinguished from one another by the identifier.

[0091] In this manner, the generation process of the teacher data for one slice image group 210 and the like are executed.

[0092] Furthermore, such a generation process of teacher data for one slice image group 210 is executed for a plurality of slice image groups 210. For example, it is executed for a plurality of slice image groups 210 related to a plurality of subjects. As a result, a very large amount of teacher data (for example, tens of thousands to hundreds of thousands) is generated.

[0093] These training data are used in the next stage (learning stage).

[0094] <1-3. Learning stage using teacher data> In addition, the controller 31 uses data reflecting annotations (such as the 3D annotation 53) for the three-dimensional surface data as training data to perform machine learning to generate a trained model. Hereinafter, such a learning stage process will be described.

[0095] Fig. 6 is a flowchart showing the processing of the controller 31 in the learning stage (see Fig. 3). Fig. 6 also shows (a part of) the method for generating a trained model.

[0096] In step S21, the controller 31 executes machine learning using the teacher data (teacher data having a known lesion site (fracture site)) generated by the process of Fig. 5. In detail, data based on the two-dimensional slice image 220 (260) to which the 2D annotation 52 is added (particularly, image data of the fracture region surrounded by the bounding box 55) is used as the teacher data to execute machine learning. As the machine learning, deep learning or the like is executed.

[0097] By such machine learning, various parameters of the learning model 410 (see FIG. 3) are adjusted, and a trained model 420 is generated (step S22). Specifically, the learning parameters (weighting coefficients between multiple layers (input layer, (one or more) intermediate layers, output layer) and the like) of the learning model 410 (learning device) are adjusted, and the trained model 420 is generated. This trained model 420 is a learning model for executing an inference process to identify a lesion site (fracture site) of an unknown two-dimensional slice image.

[0098] In this application, generating the trained model 420 means manufacturing (producing) the trained model 420, and the "method of generating a trained model" means the "method of manufacturing a trained model."

[0099] <1-4. Inference stage> Furthermore, the controller 31 performs inference processing in machine learning using a trained model 420 generated by machine learning.

[0100] Specifically, the controller 31 uses the trained model 420 to execute an inference process related to fracture detection for each of a plurality of (unknown) two-dimensional slice images 220 (also referred to as 270 (see FIG. 4)) acquired for a certain subject. In particular, an inference process is executed to identify a lesion site (fracture site) in each slice image using the trained model 420. The inference process will be described below with reference to FIG. 7.

[0101] First, when the two-dimensional slice image 270 is input to the trained model 420 as inference target data in step S31 of Fig. 7, the trained model 420 acquires an inference result regarding the input data in step S32. Specifically, the trained model 420 (controller 31) detects the presence or absence and position of a fracture site in the slice image 270. Such processing is repeatedly executed for the multiple two-dimensional slice images 270 until it is determined in step S33 that the processing should be terminated. Specifically, the image number i is incremented (step S34), and the processing is executed for a new two-dimensional slice image Mi (270).

[0102] When the trained model 420 (controller 31) detects the presence of a fracture site in the two-dimensional slice image 270, it outputs the two-dimensional slice image 270 (also referred to as 280) in which the position of the fracture site is specified in step S35 (see FIG. 4). In the slice image 280, for example, the area indicating the fracture site is displayed surrounded by a bounding box 58. In addition, the reliability (for example, 80%) of the inference result of the fracture site is also output.

[0103] Note that, although the inference results for the plurality of two-dimensional slice images 220 are collectively displayed here (step S35), the present invention is not limited to this. For example, the inference results for each of the two-dimensional slice images 220 may be displayed one by one.

[0104] <1-5. Effects of the First Embodiment> According to the above-described embodiment, the three-dimensional surface data 330 is generated based on the multiple two-dimensional slice images 220 (S12 (see FIG. 5)), and an operation of adding a 3D annotation to the three-dimensional surface data 330 is accepted (S13). Therefore, it is possible to reduce the complexity of the annotation operation compared to a case where an operation of adding an annotation to each of a large number (e.g., several hundred) of two-dimensional slice images is performed by an operating user for each two-dimensional slice image.

[0105] Furthermore, machine learning is performed using data reflecting annotations (such as the 3D annotation 53) for the three-dimensional surface data 330 as training data (S21). Therefore, it is possible to create training data while reducing the complexity of the annotation assignment operation, and to perform machine learning based on the training data.

[0106] In particular, a first process (S13) of adding a 3D annotation 53 to the three-dimensional surface data 330 in response to an operation by an operating user, and a second process (S14) subsequent to the first process are executed. The second process is a process of automatically adding a 2D annotation in at least one slice image of the multiple two-dimensional slice images 220 based on the 3D annotation 53. The second process includes a process of automatically adding a 2D annotation 52 to a two-dimensional position (corresponding two-dimensional position) corresponding to the three-dimensional position of the 3D annotation 53 in at least one two-dimensional slice image corresponding to the three-dimensional position. Then, machine learning is executed using data based on the at least one two-dimensional slice image to which the 2D annotation 52 is added as training data (S21).

[0107] This makes it possible to generate teacher data with significantly reduced complexity in annotation operations compared to creating teacher data by individually annotating two-dimensional slice images 220 (especially a large number of two-dimensional slice images 220). In particular, by executing the above-mentioned processing on a plurality of slice image groups 210, it is possible to generate a very large number of teacher data (for example, several thousand to several hundred thousand) with significantly reduced complexity in annotation operations.

[0108] Furthermore, when (2D) annotations and bounding boxes are individually added to a plurality of two-dimensional slice images 220 by an operator as in the conventional method, annotations, etc. may be added to a rough position. Specifically, annotations, etc. may be added to each two-dimensional slice image 220 without checking the positional relationship with annotations in other two-dimensional slice images 220. In this case, variation in annotation addition positions between the plurality of two-dimensional slice images 220 (addition position error due to user operation) is likely to occur.

[0109] On the other hand, in the above embodiment, the 3D annotation 53 is added based on the annotation line 51 drawn by the operating user while checking the three-dimensional position using the three-dimensional surface data 330, and the 2D annotation 52 is added based on the 3D annotation 53. More specifically, the 3D annotation 53 and the 2D annotation 52 are added based on the continuous annotation line 51 across the multiple two-dimensional slice images 220. Therefore, the annotation adding position error (variation in the adding position in the user's operation) between the multiple two-dimensional slice images 220 can be reduced. Therefore, it is possible to add the 2D annotation 52 to each two-dimensional slice image 220 more accurately.

[0110] <2. Second embodiment> <Summary> In the above first embodiment, an example is shown in which the 2D annotation 52 is added only to a two-dimensional position (corresponding two-dimensional position) in the two-dimensional slice image 220 that corresponds to the intersection line with the 3D annotation 53. In other words, in at least one slice image (corresponding to the three-dimensional position of the 3D annotation) among the multiple two-dimensional slice images 220, the 2D annotation 52 is added only to a two-dimensional position (corresponding two-dimensional position) that corresponds to the three-dimensional position of the 3D annotation 53.

[0111] 16 (and 17), when the intersection lines between a 2D slice image 220 and a 3D annotation 53 are found at two points in a 2D slice image 220, 2D annotations 52 are added to the two points. Then, a bounding box 55 that includes the two 2D annotations 52 is drawn, and a rectangular area enclosed by the bounding box 55 is determined as teacher data indicating a fracture area (see the top row of FIG. 16 and the top row of FIG. 17).

[0112] However, as will be described later, in a certain two-dimensional slice image 220 (slice cross section), the intersection line with the 3D annotation 53 may appear in only one place (see FIG. 18, etc.). In this case, in the first embodiment, a bounding box 55 is formed so as to surround only the one 2D annotation 52. As a result, a partial region including a fracture site (fracture line, etc.) (a partial region that is originally desired to be included in the training data) protrudes from the bounding box 55. In other words, the bounding box 55 cannot necessarily surround the partial region accurately.

[0113] Therefore, in the second embodiment, a technique for supplementarily adding a 2D annotation 52 to a cross section where only one intersection with the 3D annotation 53 appears (a cross section to which only a single 2D annotation 52 is added) will be described. In detail, in each two-dimensional slice image, the 2D annotation 52 is also supplementarily added to a two-dimensional position determined to be a "corresponding two-dimensional position" in any of a predetermined number of two-dimensional slice images in the vicinity of each two-dimensional slice image. In other words, in each two-dimensional slice image, the 2D annotation 52 is also supplementarily added to a two-dimensional position to which a 2D basic annotation 52A (described next) is added in any of a predetermined number of two-dimensional slice images in the vicinity of each two-dimensional slice image.

[0114] Here, the 2D annotation 52 that is assigned to the position of the line of intersection with the 3D annotation 53 in a two-dimensional slice image having the line of intersection with the 3D annotation 53 is also referred to as a "2D basic annotation" (52A). In other words, the 2D basic annotation 52A is a 2D annotation 52 that is assigned to a two-dimensional position (corresponding two-dimensional position) corresponding to the three-dimensional position of the 3D annotation 53 in at least one two-dimensional slice image corresponding to the three-dimensional position of the 3D annotation 53. The 2D annotation 52 assigned in the above first embodiment corresponds to the "2D basic annotation."

[0115] In addition, the 2D annotation 52 that is added in a complementary manner to a 2D position determined to be a "corresponding 2D position" in any of a predetermined number of 2D slice images in the vicinity of each 2D slice image is also referred to as a "2D complementary annotation" (52B). The 2D complementary annotation 52B (see FIG. 25) is a 2D annotation 52 that is added in a complementary manner separately from the 2D basic annotation 52A in the second embodiment. The 2D complementary annotation 52B is also expressed as a 2D annotation that complements the 2D basic annotation 52A.

[0116] The 2D annotation 52 including both the 2D basic annotation 52A and the 2D supplemental annotation 52B is also referred to as a "2D extended annotation" (52C) ​​(since it is a conceptual extension of the 2D basic annotation 52A). The 2D extended annotation 52C is also expressed as a 2D annotation that is assigned not only to a two-dimensional position (corresponding two-dimensional position) corresponding to the three-dimensional position of the 3D annotation 53, but also to a position other than the corresponding two-dimensional position.

[0117] The second embodiment is a modification of the first embodiment. The following description will focus on the differences from the first embodiment.

[0118] In the second embodiment, the same processes as those in FIGS. 5 to 7 are performed. However, the process in step S14 in FIG. 5 is different from that in the first embodiment. FIG. 22 is a flowchart showing the process in step S14 (also referred to as S14B) in the second embodiment. In step S51 in FIG. 22, a process similar to the process of adding a 2D annotation (2D basic annotation 52A) in step S14 in the first embodiment is performed. In the second embodiment, the process in step S52 (the process of adding a 2D complementary annotation 52B) is additionally performed, and in step S53, a bounding box is set based on the processing result of step S52, and teacher data is generated.

[0119] <Multiple 2D slice images according to the second embodiment> Before describing the processing of Fig. 22, a situation in which only one intersection line with the 3D annotation 53 appears in a certain two-dimensional slice image 220 (slice cross section) and the processing of Fig. 5 will be described below. Then, the processing of Fig. 22 will be described.

[0120] Fig. 23 is a schematic diagram showing a series of two-dimensional slice images 220. Fig. 23 shows some (seven) two-dimensional slice images 220 (also referred to as cross-sectional images L5 to L11) of the multiple two-dimensional slice images 220 for teacher data according to the second embodiment. The circular region (region with sandy hatching) in Fig. 23 represents the cross section of the bone, and the curved line therein represents the fracture line (a portion where a crack or the like in the bone appears as a linear region in the cross-sectional image).

[0121] In some (cross-sectional images L8 and L9) of the multiple cross-sectional images (two-dimensional slice images 220) in FIG. 23, the fracture line penetrates from one part (lower end part) to another part (right end part) in the circular cross section. On the other hand, in some other cross-sectional images (cross-sectional images L5 to L7, L10 to L11), the fracture line does not penetrate from one part to another in the circular cross section, but stops halfway. In detail, in the cross-sectional images L5 to L7, the fracture line extending from the lower end stops halfway without reaching the right end. Also, in the cross-sectional images L8 to L9, the fracture line extending from the right end stops halfway without reaching the lower end. Depending on the type and / or degree of the fracture, such multiple two-dimensional slice images 220 may be obtained.

[0122] 18 is a schematic diagram showing three-dimensional surface data 330 (stereoscopic model) generated based on such a two-dimensional slice image 220. The three-dimensional surface data 330 is generated in the same manner as in the first embodiment (see steps S11 and S12 (FIG. 5)).

[0123] Fig. 18 is a diagram similar to Fig. 16. In the perspective view in the middle part of Fig. 18, a slice section (two-dimensional slice image 220) at a certain Z position is also shown. The elliptical region drawn by a dashed line in the middle part indicates a portion where a cylindrical three-dimensional model intersects with the slice section. In addition, in the lower part of Fig. 18, a projection image 332 in which three-dimensional surface data 330 (three-dimensional model) is projected onto a projection plane 70 is shown. Furthermore, in the upper part (upper right part) of Fig. 18, a two-dimensional slice image 220 to which a 2D annotation 52 (specifically, a 2D basic annotation 52A) is added is typically shown.

[0124] 18, however, a curve different from the annotation line 51 in FIG. 16 is drawn as the annotation line 51. More specifically, two annotation lines 51 separated midway are drawn, rather than a single continuous annotation line 51 that goes halfway or fully around the bone (schematically represented in a cylindrical shape here). If the fracture line does not reach a certain surface portion of the bone, no part indicating a fracture appears on that surface portion (the surface of the three-dimensional surface data 330 (the surface of the three-dimensional model of the bone)), and the annotation line 51 may not be drawn on that surface portion. As a result, a situation may occur in which the annotation line 51 is separated.

[0125] One of the two annotation lines 51 is drawn from near the center to the lower left side in the projection image 332 of Fig. 18. In other words, the annotation line 51 is drawn in the projection image 332 along the fracture site that appears on the surface near the front side of the 3D surface data 330 in the middle of Fig. 18 (the lower end side of Fig. 23). Then, the 3D annotation 53 corresponding to the annotation line 51 is added along the fracture site that appears on the surface near the front side of the 3D surface data 330 in the middle of Fig. 18 (the lower end side of Fig. 23).

[0126] Another annotation line 51 is drawn in the approximately vertical direction near the right end of the center in the projection image 332 of Fig. 18. In other words, the annotation line 51 is drawn in the projection image 332 along the fracture site that appears on the surface near the right side of the 3D surface data 330 in the middle of Fig. 18 (corresponding to the right end side of Fig. 23). Then, a 3D annotation 53 corresponding to the annotation line 51 is added along the fracture site that appears on the surface near the right side of the 3D surface data 330 in the middle of Fig. 18 (corresponding to the right end side of Fig. 23).

[0127] In this way, two annotation lines 51 are drawn, and two 3D annotations 53 are added accordingly (see step S13 (FIG. 5)). The existence ranges of the two 3D annotations 53 in the Z direction (reference axis direction) are different from each other while partially overlapping each other.

[0128] 19 to 21 are similar to FIG. 18. However, in FIG. 18 to FIG. 21, slice cross sections (two-dimensional slice images 220) at different positions in the reference axis direction (here, the Z-axis direction) are shown. From FIG. 18 to FIG. 21, the positions of the two-dimensional slice images 220 in the Z-axis direction gradually become higher. The two-dimensional slice images 220 in FIG. 18 to FIG. 21 correspond to the cross-sectional images L7 to L10 in FIG. 23, etc., respectively.

[0129] Next, based on the three-dimensional surface data 330 to which such 3D annotations 53 are added, 2D annotations 52 (more specifically, 2D basic annotations 52A) are added to the intersections with the 3D annotations 53 for each of the two-dimensional slice images 220 in the same manner as in the first embodiment (see step S14) (see FIG. 24). FIG. 24 is a diagram showing a state in which the 2D basic annotations 52A are added to the series of two-dimensional slice images in FIG. 23, respectively.

[0130] 19 and 20, in a part of the two-dimensional slice images 220, two intersections with the 3D annotation 53 appear. In the part of the two-dimensional slice images 220, two 2D annotations 52 are added at different positions in the two-dimensional slice image 220.

[0131] 18 and 21, in some other two-dimensional slice images 220, an intersection line with the 3D annotation 53 appears at only one location. In the other two-dimensional slice images 220, a single 2D annotation 52 (2D basic annotation 52A) is added within the two-dimensional slice image 220.

[0132] Here, as described above, the 2D annotation 52 added by the same process as in the first embodiment is also referred to as a 2D basic annotation 52A, and is distinguished from the 2D complementary annotation 52B added complementarily in the second embodiment. In step S51 in FIG. 22, a process of adding the 2D basic annotation 52A is executed (see also FIG. 24). Specifically, in each two-dimensional slice image 220 (the i-th two-dimensional slice image Li), the 2D basic annotation 52A is added to the intersection with the 3D annotation 53. Note that FIG. 24 is a diagram showing a series of two-dimensional slice images 220 to which the 2D basic annotation 52A is added.

[0133] In such a situation, as described above, in a certain two-dimensional slice image 220 (cross-sectional images L5 to L7, L10 to L11), the intersection line with the 3D annotation 53 appears at only one location. In this case, as in the first embodiment, a bounding box 55 may be formed so as to surround the 2D annotation 52 (2D basic annotation 52A) that is added at only that one location. However, a partial region including a fracture site (a partial region that is originally desired to be included in the training data) protrudes from the bounding box 55, and the bounding box 55 cannot necessarily surround the partial region accurately.

[0134] <Addition of 2D complementary annotation 52B, etc.> Therefore, in the second embodiment, after a process of adding a 2D basic annotation 52A (step S51) is executed in the same manner as in the first embodiment, a process of adding a 2D complementary annotation 52B (step S52) is further executed.

[0135] More specifically, in step S51, in the i-th two-dimensional slice image (cross-sectional image) Li, a 2D basic annotation 52A is added to the intersection with the 3D annotation 53. Then, in step S52, in the i-th two-dimensional slice image Li, a 2D complementary annotation 52B is added (as necessary). Specifically, in the i-th two-dimensional slice image Li, a 2D complementary annotation 52B is added to a two-dimensional position to which the 2D basic annotation 52A is added in any of a predetermined number (predetermined number) of two-dimensional slice images in the vicinity of the i-th two-dimensional slice image Li. In other words, in the i-th two-dimensional slice image Li, a 2D complementary annotation 52B is added to a two-dimensional position determined to be a "corresponding two-dimensional position" (intersection with the 3D annotation 53) in any of a predetermined number of two-dimensional slice images in the vicinity of the i-th two-dimensional slice image Li.

[0136] The predetermined number is, for example, several to several tens of images. The predetermined number of two-dimensional slice images is, for example, (2*N) (where N is a natural number) two-dimensional slice images L(iN) to L(i-1), L(i+1) to L(i+N). More specifically, in the case of N=3, when i=8, in the two-dimensional slice image L8, the 2D complementary annotation 52B is added to the two-dimensional position to which the 2D basic annotation 52A is added in the six neighboring two-dimensional slice images L5 to L7, L9 to L11. In the case of N=3, when i=5, in the two-dimensional slice image L8, the 2D complementary annotation 52B is added to the two-dimensional position to which the 2D basic annotation 52A is added in the six neighboring two-dimensional slice images L2 to L4, L6 to L8. In the following description, it is assumed that the predetermined number is three images each (six images in total).

[0137] Fig. 25 is a diagram showing a plurality of two-dimensional slice images before and after the process of step S52. As in Fig. 24, the upper part of Fig. 25 shows the two-dimensional slice images 220 (cross-sectional images L5 to L11) to which only the 2D basic annotation 52A has been added (at step S51). The lower part of Fig. 25 shows the two-dimensional slice images 220 to which the 2D complementary annotation 52B has been added. In other words, the two-dimensional slice images 220 in the lower part of Fig. 25 are added with the 2D extended annotation 52C (the 2D annotation 52 including both the 2D basic annotation 52A and the 2D complementary annotation 52B).

[0138] As shown in FIG. 25, in step S52, the controller 31 complementarily adds a 2D complementary annotation 52B to each 2D slice image 220 (such as a 2D slice image to which a 2D basic annotation 52A has been added) (see the lower part of FIG. 25).

[0139] As can be seen by comparing the lower and upper rows of Figure 25, the 2D complementary annotation 52B is (automatically) added to each two-dimensional slice image 220 at a two-dimensional position to which the 2D basic annotation 52A has been added in any of a specified number (predetermined number) of slice images in the vicinity of each two-dimensional slice image 220.

[0140] For example, in the cross-sectional image L5 (two-dimensional slice image 220), in addition to the 2D basic annotation 52A near the bottom end, a 2D supplemental annotation 52B is added (near the right end).

[0141] A total of six cross-sectional images L2 to L4 and L6 to L8 near the cross-sectional image L5 have already been assigned (at step S51) 2D basic annotations 52A (see the upper part of FIG. 25). At step S52, all of the 2D basic annotations 52A assigned to these cross-sectional images L2 to L4 and L6 to L8 are assigned as 2D complementary annotations 52B to corresponding positions in the cross-sectional image L5.

[0142] Specifically, when the 2D basic annotation 52A is added near the right end of the nearby cross-sectional image L8, the 2D complementary annotation 52B is further added to a position near the right end of the cross-sectional image L5 (the position where the 2D basic annotation 52A is added in the nearby cross-sectional image L8). In other words, in the cross-sectional image L5, the 2D complementary annotation 52B is added complementarily to a two-dimensional position (near the right end) different from the position (near the bottom end) of the 2D basic annotation 52A of the cross-sectional image L5 (see the lower part of FIG. 25). Note that in FIG. 25, the 2D complementary annotation 52B added at approximately the same position as the 2D basic annotation 52A is omitted as appropriate. For example, the 2D complementary annotation 52B added near the bottom end in the cross-sectional image L5 (the 2D complementary annotation 52B added in response to the 2D basic annotation 52A added near the bottom end in the nearby cross-sectional image L6) is omitted.

[0143] The same is true for the cross-sectional images L6 and L7. Specifically, in each of the cross-sectional images L6 and L7, a 2D complementary annotation 52B is added to a position near the right end of the cross-sectional image L6 and L7 (a position where the 2D basic annotation 52A is added in the neighboring cross-sectional image L8).

[0144] In addition to the 2D basic annotation 52A near the right end of the cross-sectional image L10, a 2D complementary annotation 52B is added (near the bottom end, etc.). The 2D basic annotation 52A has already been added (at step S51) to each of the three cross-sectional images L7 to L9 and L11 to L13 near the cross-sectional image L10 (see the upper part of FIG. 25). In step S52, all of the 2D basic annotations 52A added to these cross-sectional images L7 to L9 and L11 to L13 are added to corresponding positions in the cross-sectional image L5 as 2D complementary annotations 52B. As a result, the 2D complementary annotation 52B is added especially near the bottom end of the cross-sectional image L10 (see the lower part of FIG. 25).

[0145] The same is true for the cross-sectional image L11. The 2D complementary annotation 52B is added to a position near the bottom end of the cross-sectional image L11 (the position where the 2D basic annotation 52A is added in the cross-sectional images L8 and L9).

[0146] The same is true for the other cross-sectional images Li. In each cross-sectional image Li, a 2D supplemental annotation 52B is added to a position (corresponding position) to which a 2D basic annotation 52A is added in a predetermined number of neighboring cross-sectional images (two-dimensional slice images 220). In other words, the 2D annotation 52 is supplemented.

[0147] In this manner, in step S51, in the i-th two-dimensional slice image Li, the 2D basic annotation 52A is added to the intersection with the 3D annotation 53. Then, in step S52, in the i-th two-dimensional slice image Li, the 2D complementary annotation 52B is added to a two-dimensional position determined to be a "corresponding two-dimensional position" (intersection with the 3D annotation 53) in any of a predetermined number of two-dimensional slice images (L(iN) to L(i-1), L(i+1) to L(i+N)) in the vicinity of the i-th two-dimensional slice image Li. In other words, the 2D complementary annotation 52B is added to a certain slice image in a complementary manner using the 2D basic annotation 52A added to a predetermined number of slice images in the vicinity of the certain slice image.

[0148] As a result, in steps S51 and S52, the 2D annotation 52 (2D extended annotation 52C) is added as follows: The 2D annotation 52 is added to all two-dimensional positions in each two-dimensional slice image Li that are determined to be two-dimensional positions (corresponding two-dimensional positions) corresponding to the three-dimensional positions of the 3D annotation in either the two-dimensional slice image Li or a predetermined number of nearby two-dimensional slice images.

[0149] More specifically, in the i-th two-dimensional slice image Li, the 2D extended annotation 52C is added to all two-dimensional positions determined to be "corresponding two-dimensional positions" (intersections with the 3D annotation 53) in any of two-dimensional slice images in a predetermined range (for example, (L(iN) to L(i-1), Li, L(i+1) to L(i+N))) that include the i-th two-dimensional slice image Li. In other words, in the i-th two-dimensional slice image Li, the 2D extended annotation 52C is added as a logical OR set of the 2D basic annotations 52A in a total of (2*N+1) two-dimensional slice images in a neighboring range that includes the i-th two-dimensional slice image Li.

[0150] Thereafter, in step S53, a bounding box 55 is formed according to both the 2D basic annotation 52A and the 2D supplemental annotation 52B. The bounding box 55 is formed as a circumscribing rectangle surrounding both. Then, a partial image within the bounding box 55 is identified as a partial image including a fracture site. In other words, a partial image within at least a portion of two-dimensional slice images, each of which has two or more 2D annotations added thereto, among a plurality of two-dimensional slice images, a partial image within a bounding box according to the two or more 2D annotations is identified as a partial image including a fracture site.

[0151] As a result, as shown in the lower part of Fig. 25, the bounding boxes 55 are formed to appropriately surround the fractured portion (fracture line, etc.) not only in the cross-sectional images L8 and L9 but also in the cross-sectional images L5 to L7, L10, and L11. In other words, appropriate partial images that do not include the fractured portion but do not protrude from the fractured portion are extracted as training data.

[0152] Furthermore, machine learning is executed using the partial image as training data (see step S21 in FIG. 6, etc.). Through the machine learning, a trained model 420 is generated (see also FIG. 3).

[0153] In addition, inference processing in machine learning is performed using a trained model 420 generated by machine learning (see FIGS. 4 and 7).

[0154] In the above-described aspect, the 2D annotation 52 is assigned to a two-dimensional position (corresponding two-dimensional position) corresponding to the three-dimensional position of the 3D annotation in at least one slice image having an intersection line with the 3D annotation 53 among the multiple two-dimensional slice images 220. Furthermore, the 2D annotation 52 is also assigned to a two-dimensional position in each two-dimensional slice image that is determined to be the corresponding two-dimensional position in any of a predetermined number of two-dimensional slice images in the vicinity of each two-dimensional slice image. That is, both the 2D basic annotation 52A and the 2D supplemental annotation 52B are assigned as the 2D annotation 52. In other words, the 2D extended annotation 52C is assigned as the 2D annotation 52.

[0155] According to this embodiment, the 2D complementary annotation 52B is also added. Therefore, it is possible to add a more appropriate 2D annotation 52 compared to the case where only the 2D basic annotation 52A is added. Furthermore, it is possible to generate more appropriate teacher data. Furthermore, by performing machine learning using such appropriate teacher data, it is possible to optimize the parameters in the learning model, in other words, to improve the accuracy of the inference process using the learned model.

[0156] <3. Modifications, etc.> Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described contents.

[0157] For example, in each of the above embodiments, the 3D annotation 53 has a predetermined width (is a long and narrow planar region), and the 2D annotation 52 is added to the intersection portion between each two-dimensional slice image 220 and the 3D annotation 53, but is not limited to this. Specifically, the 3D annotation 53 may be linear, and the 2D annotation 52 may be added to the intersection portion between each two-dimensional slice image 220 and the 3D annotation 53. In other words, the two-dimensional position corresponding to the three-dimensional position of the 3D annotation 53 may be the intersection portion between the two-dimensional slice image 220 and the 3D annotation, or may be the intersection portion between the two-dimensional slice image 220 and the 3D annotation, etc.

[0158] In addition, in each of the above-mentioned embodiments, the 3D annotation 53 is added based on the drawing operation of the annotation line 51 (an elongated planar region having a predetermined width), but the present invention is not limited to this. For example, instead of or together with the drawing operation of the annotation line 51 in the projection image 332 (see FIG. 14 and FIG. 15, etc.), the 3D annotation 53 may be added based on the drawing operation of a closed curve (circle, ellipse, polygon, etc.) in the projection image 332. In detail, a polygon in a closed region surrounded by the closed curve (including a closed line) may be specified as indicating a fracture site. Then, the three-dimensional position of each polygon may be specified as the three-dimensional position of the 3D annotation 53. Furthermore, the 2D annotation 52 may be added to the intersection portion between the 3D annotation 53 (a region having a planar spread) corresponding to the drawing operation of such a closed region and each two-dimensional slice image 220.

[0159] In the second embodiment, both the 2D basic annotations and the 2D complementary annotations are sequentially added (S51, S52). In detail, a process of adding the 2D basic annotations 52A (step S51) is first (preliminarily) performed on a plurality of 2D slice images 220, and then a process of supplementing each 2D slice image with the 2D complementary annotations 52B (step S52) is performed. However, the present invention is not limited to this, and both annotations may be added simultaneously in parallel.

[0160] For example, in a state where the process of step S51 is not performed, a plurality of 2D extended annotations 52C may be added at once (collectively) in each two-dimensional slice image Li.

[0161] In detail, the controller 31 judges whether or not a "corresponding two-dimensional position" (intersection with the 3D annotation 53) exists in a predetermined range of two-dimensional slice images (for example, (L(iN) to L(i-1), Li, L(i+1) to L(i+N))) including the i-th two-dimensional slice image Li. Then, the controller 31 assigns the 2D extended annotation 52C to all two-dimensional positions (positions in the i-th two-dimensional slice image Li) judged to be "corresponding two-dimensional positions" in any of the two-dimensional slice images in the predetermined range (neighborhood range). Furthermore, the controller 31 repeatedly executes the above-mentioned process for all i (all two-dimensional slice images 220).

[0162] In this manner, for each 2D slice image Li, the "corresponding 2D positions" in the 2D slice images in the vicinity including the 2D slice image Li itself may be obtained, and the 2D annotation 52 (2D extended annotation 52C) consisting of the logical sum of the obtained positions may be collectively added. In other words, the 2D extended annotation 52C may be added by a method different from the method of adding the 2D basic annotation 52A to the multiple 2D slice images 220 in advance and then adding the 2D complementary annotation 52B.

[0163] In this manner, too, a 2D extended annotation 52C including a 2D basic annotation 52A and a 2D complementary annotation 52B is added.

[0164] In the second embodiment and the like, a logical sum (logical sum of corresponding two-dimensional positions) is obtained over a range of a plurality of two-dimensional slice images having the same number (N pieces) before and after each two-dimensional slice image Li. However, the present invention is not limited to this. For example, a logical sum may be obtained over a range (a range of a plurality of two-dimensional slice images) having a different number of pieces before and after each two-dimensional slice image Li. [Explanation of symbols]

[0165] 10 Image Processing System 20-slice image generator 30 Image processing device 51 Annotation Line 52 2D Annotation 52A 2D Basic Annotation 52B 2D Complementary Annotation 52C 2D Extended Annotation 53 3D Annotation 55,58 Bounding Box 70 Projection plane 210 slice image set 220,260,Li 2D slice image (cross-sectional image) (for training data) 270,Mi 2D slice image (for inference processing) 280 2D slice images (including inference results) 330,340 3D surface data 332 Projected Images 410 Learning Model 420 trained models

Claims

1. A training data generation device that generates training data for machine learning a learning model that identifies a lesion site in a target image when the target image is input, the training data generation device comprising: A control unit for generating three-dimensional surface data based on a plurality of two-dimensional slice images; a reception unit that receives an operation of adding a 3D annotation, the 3D annotation being an annotation for a lesion site in the 3D surface data and having 3D position information; Equipped with The control unit is executing a first process of adding the 3D annotation to the three-dimensional surface data in response to the adding operation; performing a second process of adding a 2D annotation, which is an annotation having two-dimensional position information, in at least one two-dimensional slice image of the plurality of two-dimensional slice images based on the 3D annotation; A training data generating device, characterized in that it generates training data having a bounding box indicating a known lesion site in the at least one two-dimensional slice image based on the 2D annotation.

2. The teacher data generating device according to claim 1 , wherein the reception unit receives an operation for adding a 3D annotation that spans a range spanning two or more of the plurality of two-dimensional slice images.

3. the reception unit receives a designation operation of a two-dimensional position on a projection surface onto which the three-dimensional surface data is projected as a designation operation of a position of the 3D annotation; The teacher data generation device according to claim 1 or claim 2, characterized in that the control unit converts a two-dimensional position on the projection surface into a three-dimensional position on the three-dimensional surface data, and generates three-dimensional position information of the 3D annotation.

4. The 2D annotation may include: a corresponding two-dimensional position in the at least one two-dimensional slice image corresponding to the three-dimensional position of the 3D annotation, the corresponding two-dimensional position being a two-dimensional position corresponding to the three-dimensional position of the 3D annotation; The teacher data generating device according to any one of claims 1 to 3, characterized in that, within each two-dimensional slice image, the corresponding two-dimensional position is also assigned to any of a predetermined number of two-dimensional slice images in the vicinity of the respective two-dimensional slice image.

5. The 2D annotation may include: a 2D basic annotation that is assigned to a two-dimensional position corresponding to the three-dimensional position of the 3D annotation in the at least one two-dimensional slice image; and a 2D complementary annotation that complements the 2D basic annotation, the 2D complementary annotation being added to each of the two-dimensional slice images at a two-dimensional position to which the 2D basic annotation is added in any of a predetermined number of slice images in the vicinity of each of the two-dimensional slice images; The teacher data generating device according to claim 1 , further comprising:

6. the control unit assigns, as the 2D annotation, a 2D extended annotation that is assigned not only to a corresponding two-dimensional position that is a two-dimensional position corresponding to the three-dimensional position of the 3D annotation but also to a position other than the corresponding two-dimensional position; The 2D extended annotation includes: a 2D basic annotation that is assigned to a two-dimensional position corresponding to the three-dimensional position of the 3D annotation in the at least one two-dimensional slice image that corresponds to the three-dimensional position of the 3D annotation; a 2D complementary annotation that is assigned to each of the two-dimensional slice images at a two-dimensional position to which the 2D basic annotation is assigned in any of a predetermined number of two-dimensional slice images in the vicinity of the respective two-dimensional slice images; The teacher data generating device according to claim 1 , further comprising:

7. The teacher data generation device according to any one of claims 1 to 3, characterized in that the 2D annotation is assigned to all two-dimensional positions within each two-dimensional slice image that are determined to be two-dimensional positions corresponding to the three-dimensional position of the 3D annotation in either the respective two-dimensional slice image or a predetermined number of adjacent two-dimensional slice images.

8. The teacher data generating device according to any one of claims 4 to 7, characterized in that the control unit generates as the teacher data a partial image within at least some of the multiple two-dimensional slice images, each of which has two or more 2D annotations added thereto, the partial image being within a bounding box corresponding to the two or more 2D annotations.

9. The teacher data generating device of claim 5 or claim 6, characterized in that the control unit generates as the teacher data a partial image within at least a portion of the multiple 2D slice images, the partial image being within a bounding box corresponding to both the 2D basic annotations and the 2D complementary annotations.

10. 8. The teaching data generating device according to claim 1, wherein the three-dimensional surface data is data representing a three-dimensional surface shape of a bone.

11. A method for generating training data for machine learning a learning model that identifies a lesion site in a target image when the target image is input, comprising: a) receiving, by a computer, an operation for adding a 3D annotation, the 3D annotation being an annotation for a lesion site in 3D surface data generated based on a plurality of 2D slice images and having 3D position information; b) the computer assigning the 3D annotation to the three-dimensional surface data in response to the assignment operation; c) adding, by the computer, a 2D annotation, which is an annotation having two-dimensional position information, in at least one two-dimensional slice image of the plurality of two-dimensional slice images based on the 3D annotation; d) generating the training data having a bounding box indicating a known lesion site in the at least one two-dimensional slice image based on the 2D annotations by the computer; A teacher data generating method comprising:

12. In step c), the 2D annotation is a corresponding two-dimensional position in the at least one slice image that corresponds to the three-dimensional position of the 3D annotation; The teacher data generating method according to claim 11, characterized in that, within each two-dimensional slice image, the corresponding two-dimensional position is also assigned to any of a predetermined number of two-dimensional slice images in the vicinity of the respective two-dimensional slice image.

13. A method for producing a trained model by machine learning a learning model that identifies a lesion site in a target image when the target image is input, comprising: a) receiving, by a computer, an operation for adding a 3D annotation, the 3D annotation being an annotation for a lesion site in 3D surface data generated based on a plurality of 2D slice images and having 3D position information; b) the computer assigning the 3D annotation to the three-dimensional surface data in response to the assignment operation; c) adding, by the computer, a 2D annotation, which is an annotation having two-dimensional position information, in at least one two-dimensional slice image of the plurality of two-dimensional slice images based on the 3D annotation; d) generating training data having bounding boxes representing known lesion sites in the at least one two-dimensional slice image based on the 2D annotations; e) the computer performs machine learning of the learning model using the training data regarding the at least one two-dimensional slice image to generate the learned model; A method for manufacturing a trained model, comprising:

14. In step c), the 2D annotation is a corresponding two-dimensional position in the at least one slice image that corresponds to the three-dimensional position of the 3D annotation; In each two-dimensional slice image, the position is also assigned to a two-dimensional position determined to be the corresponding two-dimensional position in any of a predetermined number of two-dimensional slice images in the vicinity of the respective two-dimensional slice image, The step e) comprises: e-1) performing machine learning using partial images in at least some of the two-dimensional slice images, each of which has two or more 2D annotations added thereto, within bounding boxes corresponding to the two or more 2D annotations, as the training data; The method for producing a trained model according to claim 13, further comprising:

15. A program for causing a computer to execute a process of generating teacher data for machine learning a learning model that identifies a lesion site in a target image when the target image is input, the program comprising: a) receiving an operation for adding a 3D annotation, the 3D annotation being an annotation for a lesion site in 3D surface data generated based on a plurality of 2D slice images and having 3D position information; b) adding the 3D annotation to the three-dimensional surface data in response to the adding operation; c) adding a 2D annotation, which is an annotation having two-dimensional position information, in at least one two-dimensional slice image of the plurality of two-dimensional slice images based on the 3D annotation; d) generating the training data having bounding boxes representing known lesion sites in the at least one two-dimensional slice image based on the 2D annotations; A program for causing the computer to execute the above.

16. A program for causing a computer to execute a process of machine learning a learning model for identifying a lesion site in a target image when the target image is input, a) receiving an operation for adding a 3D annotation, the 3D annotation being an annotation for a lesion site in 3D surface data generated based on a plurality of 2D slice images and having 3D position information; b) adding the 3D annotation to the three-dimensional surface data in response to the adding operation; c) adding a 2D annotation, which is an annotation having two-dimensional position information, in at least one two-dimensional slice image of the plurality of two-dimensional slice images based on the 3D annotation; d) generating training data having bounding boxes representing known lesion sites in the at least one two-dimensional slice image based on the 2D annotations; e) machine-learning the learning model using the training data regarding the at least one two-dimensional slice image; A program for causing the computer to execute the above.

17. The step a) comprises: receiving a designation operation of a two-dimensional position on a projection surface onto which the three-dimensional surface data is projected as a designation operation of a position of the 3D annotation; Equipped with The step b) comprises: converting the two-dimensional position on the projection surface into a three-dimensional position on the three-dimensional surface data, and generating three-dimensional position information of the 3D annotation; The program according to claim 15 or 16, comprising:

Citation Information

Patent Citations

  • A method for detecting polyps in a 3D image volume

    JP2008515595A

  • Surgery supporting system and display method

    JP2019162339A

  • Automated cardiac volume segmentation

    JP2019504659A

  • Three-dimensional medical image analysis method and system for identification of vertebral fractures

    WO2019106061A1