Image processing device, method of operating the image processing device, operating program of the image processing device

Abstract

This image processing device is provided with a processor and a memory connected to or built into the processor. The processor uses a semantic segmentation model trained by using an annotation image in which: first pixels corresponding to one point corresponding to an object, a plurality of points that are discrete corresponding to a plurality of objects, and / or a line corresponding to an object that has a line structure are a first pixel value, and second pixels other than the foregoing are a second pixel value different from the first pixel value. The semantic segmentation model is trained by allocating a greater weight to the first pixels than the second pixels and computing loss. An image to be analyzed is input to the semantic segmentation model; a feature amount map that has feature amounts associated with one point, a plurality of points that are discrete, and / or a line in the image to be analyzed is output from the semantic segmentation model; and on the basis of the feature amount map, the one point, the plurality of points that are discrete, and / or the line in the image to be analyzed is identified.

Description

[Technical Field]

[0001] The technology disclosed herein includes an image processing device, a method for operating an image processing device, and an operating program for an image processing device. Mu To relate to. [Background technology]

[0002] As a machine learning model for handling images, convolutional neural networks (hereinafter abbreviated as CNNs) that perform semantic segmentation to identify objects in the image being analyzed on a pixel-by-pixel basis are well known. For example, Japanese Patent Publication No. 2020-025730 describes using a CNN to identify multiple objects in a radiation image obtained by irradiating a patient with radiation, which is used as the image to be analyzed. Examples of objects include the lungs, spine (backbone), and other regions, and it is also described that the thoracic and lumbar vertebrae of the spine can be identified separately. [Overview of the project] [Problems that the invention aims to solve]

[0003] In CNN-based semantic segmentation, objects of a certain size, such as lung fields or the spine, can be identified with relatively high accuracy. However, it has been unable to accurately identify minute objects, such as the center point of a vertebral body, the center point of the left and right eyeballs, or the midline of the aorta.

[0004] One embodiment of the technology of this disclosure is an image processing device capable of accurately identifying minute objects, a method for operating the image processing device, and an operating program for the image processing device. Mu provide. [Means for solving the problem]

[0005] The image processing apparatus of the present disclosure comprises a processor and memory connected to or embedded in the processor, wherein the processor is a semantic segmentation model learned using an annotation image in which a first pixel corresponding to at least one of a single point corresponding to an object, a discrete set of points corresponding to multiple objects, and a line corresponding to an object having a line structure is assigned a first pixel value, and other second pixels are assigned second pixel values ​​different from the first pixel value, and the semantic segmentation model learned by assigning a larger weight to the first pixel than to the second pixel and calculating the loss, inputs an image to be analyzed into the semantic segmentation model, causes the semantic segmentation model to output a feature map having features related to at least one of a single point, a discrete set of points, and a line in the image to be analyzed, and identifies at least one of a single point, a discrete set of points, and a line in the image to be analyzed based on the feature map.

[0006] The processor preferably identifies the centroid of the feature map based on the features as a single point.

[0007] The feature map is a probability distribution map whose features are the probability of existence of multiple discrete points. Preferably, the processor generates an output image in which each pixel is labeled with a class corresponding to the probability of existence in the probability distribution map, and identifies multiple discrete points based on the output image.

[0008] The feature map is a probability distribution map whose features are the probability of existence of a discrete set of points. Preferably, the processor selects elements from each element of the probability distribution map that have a probability of existence above a predetermined threshold as candidates for the discrete set of points, assigns a predetermined rectangular frame to the selected candidates, applies a non-maximal suppression process to the rectangular frame, and identifies the discrete set of points based on the results of the non-maximal suppression process.

[0009] The feature map is a probability distribution map whose features are the probability of a line existing. Preferably, the processor generates an output image in which each pixel is labeled with a class corresponding to the probability of existence in the probability distribution map, applies a thinning process to the output image, and identifies the lines based on the results of the thinning process.

[0010] The feature map is a probability distribution map whose features are the probability of a line existing. Preferably, the processor selects elements from the probability distribution map that have an existence probability above a predetermined threshold as candidates for a line, assigns a rectangular frame of a predetermined size to the selected candidates, applies a non-maximal suppression process to the rectangular frame, and identifies the line based on the results of the non-maximal suppression process.

[0011] The images to be analyzed are medical images depicting the inside of a patient's body, and preferably the objects are structures of the body.

[0012] The method of operating the image processing apparatus of the present disclosure includes: using a semantic segmentation model trained using an annotation image in which a first pixel corresponding to at least one of a single point corresponding to an object, a discrete set of points corresponding to multiple objects, and a line corresponding to an object having a line structure is assigned a first pixel value, and other second pixels are assigned second pixel values ​​different from the first pixel value; using a semantic segmentation model trained by assigning a larger weight to the first pixel than to the second pixel and calculating the loss; inputting an image to be analyzed into the semantic segmentation model and causing the semantic segmentation model to output a feature map having features related to at least one of a single point, a discrete set of points, and a line in the image to be analyzed; and identifying at least one of a single point, a discrete set of points, and a line in the image to be analyzed based on the feature map.

[0013] The operating program for the image processing apparatus of this disclosure is a semantic segmentation model learned using an annotation image in which a first pixel corresponding to at least one of a single point corresponding to an object, a discrete set of points corresponding to multiple objects, and a line corresponding to an object having a line structure is assigned a first pixel value, and other second pixels are assigned second pixel values ​​different from the first pixel value. The program uses a semantic segmentation model learned by assigning a larger weight to the first pixel than to the second pixel and calculating the loss; inputs an image to be analyzed into the semantic segmentation model, causes the semantic segmentation model to output a feature map having features related to at least one of a single point, a discrete set of points, and a line in the image to be analyzed; and causes a computer to perform the following processes: input an image to be analyzed into the semantic segmentation model, causes the semantic segmentation model to output a feature map having features related to at least one of a single point, a discrete set of points, and a line in the image to be analyzed, based on the feature map.

[0014] The learning method of this disclosure includes using an annotation image in which a first pixel corresponding to at least one of a single point corresponding to an object, a discrete set of points corresponding to multiple objects, and a line corresponding to an object having a linear structure is assigned a first pixel value, and other second pixels are assigned second pixel values ​​different from the first pixel value, and calculating a loss by assigning a weight greater to the first pixel than to the second pixel, thereby training a semantic segmentation model that outputs a feature map having features associated with at least one of a single point, a discrete set of points, and a line in the image under analysis. [Effects of the Invention]

[0015] According to the technology disclosed herein, an image processing device capable of accurately identifying minute objects, a method for operating the image processing device, and an operating program for the image processing device are available. Mu It can be provided. [Brief explanation of the drawing]

[0016] [Figure 1]It is a diagram showing a medical system including a diagnostic support device. [Figure 2] It is a block diagram showing a computer constituting a diagnostic support device. [Figure 3] It is a block diagram showing a processing unit of a CPU of a diagnostic support device. [Figure 4] It is a diagram showing a first screen. [Figure 5] It is a diagram showing an extraction unit. [Figure 6] It is a diagram showing a probability distribution map. [Figure 7] It is a diagram showing a semantic segmentation model for point extraction. [Figure 8] It is an explanatory diagram of a convolution process performed on target data. [Figure 9] It is a diagram showing the configuration of calculation data. [Figure 10] It is an explanatory diagram of pooling processing. [Figure 11] It is a diagram showing an output unit of a semantic segmentation model for point extraction. [Figure 12] It is a diagram showing an overview of processing in the learning phase of a semantic segmentation model for point extraction. [Figure 13] It is a diagram showing non-maximum suppression processing. [Figure 14] It is a diagram showing a state of generating point position information based on a point image and generating a point position display map based on the point position information. [Figure 15] It is a diagram showing a semantic segmentation model for object identification. [Figure 16] It is an explanatory diagram of a convolution process performed on a tomographic image and a point position display map. [Figure 17] It is a diagram showing an output image. [Figure 18] It is a diagram showing an overview of processing in the learning phase of a semantic segmentation model for object identification. [Figure 19] It is a diagram showing a labeling result. [Figure 20]This is a diagram of the second screen. [Figure 21] This is a flowchart showing the processing procedure of the diagnostic support device. [Figure 22] This is a diagram showing the extraction unit of the second embodiment. [Figure 23] This figure shows a semantic segmentation model for point extraction according to the second embodiment. [Figure 24] This figure shows the output section of the semantic segmentation model for point extraction according to the second embodiment. [Figure 25] This figure shows how point position information is generated based on the output image. [Figure 26] This is a diagram showing the extraction unit of the third embodiment. [Figure 27] This figure shows how point location information is generated based on a probability distribution map. [Figure 28] This figure shows another example of the extraction unit of the third embodiment. [Figure 29] This is a diagram showing the extraction unit of the fourth embodiment. [Figure 30] This figure shows a semantic segmentation model for line extraction according to the fourth embodiment. [Figure 31] This figure shows the output section of the line extraction semantic segmentation model of the fourth embodiment. [Figure 32] This figure shows how line position information is generated based on the output image. [Figure 33] This figure shows another example of the extraction unit of the fourth embodiment. [Figure 34] This figure shows the non-maximum suppression process. [Figure 35] This figure shows the process of non-maximum suppression and the generation of line position information based on the line image. [Modes for carrying out the invention]

[0017] [First Embodiment] As an example, as shown in Figure 1, the medical system 2 includes a CT (Computed Tomography) device 10, a PACS (Picture Archiving and Communication System) server 11, and a diagnostic support device 12. These CT device 10, PACS server 11, and diagnostic support device 12 are connected to a LAN (Local Area Network) 13 installed within the medical facility, and can communicate with each other via the LAN 13.

[0018] As is well known, the CT scanner 10 acquires multiple projection data by radiographically imaging patient P at different projection angles, and outputs a tomographic image 15 of patient P by reconstructing the acquired multiple projection data. The tomographic image 15 is voxel data representing the three-dimensional shape of the internal structure of patient P. In this example, the tomographic image 15 is an image of the upper body of patient P (hereinafter referred to as the upper tomographic image). Figure 1 shows the upper tomographic image 15S of the sagittal section. The upper tomographic image 15 shows the spine SP, which is composed of multiple vertebrae VB. The CT scanner 10 transmits the upper tomographic image 15 to the PACS server 11. The PACS server 11 stores and manages the upper tomographic image 15 from the CT scanner 10. The upper tomographic image 15 is an example of the "image to be analyzed" and "medical image" related to the technology of this disclosure. Also, the vertebrae VB are Related to technology These are examples of "objects" and "structures." Note that the reconstruction of projection data may be performed using a diagnostic support device 12 or the like, instead of the CT scanner 10.

[0019] The diagnostic support device 12 is, for example, a desktop personal computer and is an example of an "image processing device" related to the technology disclosed herein. The diagnostic support device 12 includes a display 17 and an input device 18. The input device 18 is a keyboard, mouse, touch panel, microphone, etc. The physician operates the input device 18 to send a request to the PACS server 11 for the distribution of the upper tomography image 15 of patient P. The PACS server 11 searches for the requested upper tomography image 15 of patient P and distributes it to the diagnostic support device 12. The diagnostic support device 12 displays the upper tomography image 15 distributed from the PACS server 11 on the display 17. The physician observes the vertebral VB of patient P shown in the upper tomography image 15 and makes a diagnosis of fracture, bone metastasis of cancer, etc. Although only one CT scanner 10 and one diagnostic support device 12 are depicted in Figure 1, there may be multiple CT scanners 10 and diagnostic support devices 12.

[0020] As an example, as shown in Figure 2, the computer constituting the diagnostic support device 12 includes, in addition to the aforementioned display 17 and input device 18, storage 20, memory 21, CPU (Central Processing Unit) 22, and communication unit 23. These are interconnected via a bus line 24. The CPU 22 is an example of a "processor" related to the technology of this disclosure.

[0021] Storage 20 is a hard disk drive built into the computer constituting the diagnostic support device 12, or connected via cable or network. Alternatively, storage 20 is a disk array consisting of multiple hard disk drives installed in series. Storage 20 stores control programs such as the operating system, various application programs, and various data associated with these programs. A solid-state drive may be used instead of a hard disk drive.

[0022] Memory 21 is work memory for the CPU 22 to execute processing. The CPU 22 loads programs stored in storage 20 into memory 21 and executes processing according to the programs. In this way, the CPU 22 comprehensively controls various parts of the computer. The communication unit 23 controls the transmission of various information with external devices such as the PACS server 11. Note that memory 21 may be built into the CPU 22.

[0023] As an example, as shown in Figure 3, the storage 20 of the diagnostic support device 12 stores an operating program 30. The operating program 30 is an application program that causes the computer constituting the diagnostic support device 12 to function as an "image processing device" according to the technology of this disclosure. In other words, the operating program 30 is an example of an "operating program for an image processing device" according to the technology of this disclosure. The storage 20 also stores the upper tomographic image 15, a semantic segmentation (hereinafter abbreviated as SS) model 32 for point extraction, and an SS model 33 for object identification. The SS model 32 for point extraction is an example of a "semantic segmentation model" according to the technology of this disclosure. In addition, the storage 20 also stores the physician's findings on the vertebral VB shown in the upper tomographic image 15, and data from various screens displayed on the display 17.

[0024] When the operating program 30 is started, the CPU 22 of the computer constituting the diagnostic support device 12 works in cooperation with the memory 21 and the like to function as a read / write (hereinafter abbreviated as RW) control unit 40, an instruction reception unit 41, an extraction unit 42, a point position display map generation unit 43, an object identification unit 44, an anatomical naming unit 45, and a display control unit 46.

[0025] The RW control unit 40 controls the storage of various data to the storage 20 and the reading of various data from the storage 20. For example, the RW control unit 40 receives an upper tomographic image 15 from the PACS server 11 and stores the received upper tomographic image 15 in the storage 20. In Figure 3, only one upper tomographic image 15 is stored in the storage 20, but multiple upper tomographic images 15 may be stored in the storage 20.

[0026] The RW control unit 40 reads the upper tomographic image 15 of patient P, designated by the physician for diagnosis, from the storage 20 and outputs the read upper tomographic image 15 to the extraction unit 42, the object identification unit 44, and the display control unit 46. The RW control unit 40 also reads the point extraction SS model 32 from the storage 20 and outputs the read point extraction SS model 32 to the extraction unit 42. Furthermore, the RW control unit 40 reads the object identification SS model 33 from the storage 20 and outputs the read object identification SS model 33 to the object identification unit 44.

[0027] The instruction receiving unit 41 receives various instructions from the physician through the input device 18. Examples of instructions received by the instruction receiving unit 41 include instructions for analysis of the upper tomographic image 15, and instructions for storing findings on the vertebral vB in the storage 20.

[0028] When the instruction receiving unit 41 receives an analysis instruction, it outputs a message to the extraction unit 42 indicating that an analysis instruction has been received. Furthermore, when the instruction receiving unit 41 receives a finding storage instruction, it outputs a message to the RW control unit 40 indicating that a finding storage instruction has been received, along with the findings.

[0029] The extraction unit 42 uses the point extraction SS model 32 to extract points within each vertebra VB visible in the upper tomographic image 15. Here, the central point CP of the vertebral body (see Figure 12, etc.) is extracted as a point within the vertebral VB. The extraction unit 42 generates point position information 50 representing the location of the central point CP of the vertebral body. The extraction unit 42 outputs the point position information 50 to the point position display map generation unit 43. The central point CP of the vertebral body is an example of "a plurality of discrete points" related to the technology of this disclosure.

[0030] The point position display map generation unit 43 generates a point position display map 51 representing the position of the central point CP of the vertebral body in the upper tomographic image 15, based on the point position information 50 from the extraction unit 42. In this example, the central point CP of the vertebral body is represented by a single pixel 136 (see Figure 16) in the upper tomographic image 15. The point position display map generation unit 43 outputs the point position display map 51 to the object identification unit 44.

[0031] The object identification unit 44 identifies each individual vertebra VB based on the upper tomographic image 15 and the point location display map 51. More specifically, the object identification unit 44 inputs the upper tomographic image 15 and the point location display map 51 into the object identification SS model 33, and the object identification SS model 33 outputs an output image 52 (see also Figure 17) in which each individual vertebra VB is identified. The object identification unit 44 outputs the output image 52 to the anatomical naming unit 45.

[0032] The anatomical naming unit 45 assigns an anatomical name to each vertebra VB identified in the output image 52. The anatomical naming unit 45 outputs the assignment result 53, which is the result of assigning anatomical names to the vertebra VBs, to the display control unit 46.

[0033] The display control unit 46 controls the display of various screens on the display 17. These screens include a first screen 60 (see Figure 4) for giving analysis instructions for the upper tomographic image 15 by the extraction unit 42, the point position display map generation unit 43, the object identification unit 44, and the anatomical naming unit 45, and a second screen 155 (see Figure 20) for displaying the naming results 53.

[0034] Figure 4 shows an example of a first screen 60 for giving instructions for the analysis of the upper tomographic image 15. The first screen 60 displays, for example, a sagittal upper tomographic image 15S of patient P being diagnosed with spinal SP. Below the upper tomographic image 15S, there is a group of buttons 61 for switching the display. In addition to or instead of the sagittal upper tomographic image 15S, axial and coronal upper tomographic images 15 may also be displayed.

[0035] The first screen 60 displays a findings input field 62, a message 63, an OK button 64, and an analysis button 65. The physician enters findings for vertebral VB into the findings input field 62. After entering the findings into the findings input field 62, the physician selects the OK button 64 by moving the cursor 66 over it. This allows the instruction receiving unit 41 to receive the finding storage instruction. The RW control unit 40 associates the upper tomographic image 15 with the findings entered in the findings input field 62 and stores them in the storage 20.

[0036] Message 63 prompts the user to select the analysis button 65. If the physician wishes to analyze the upper tomographic image 15 before entering findings, they move the cursor 66 to the analysis button 65 and select it. As a result, the instruction receiving unit 41 receives the instruction to analyze the upper tomographic image 15 and outputs a message to the extraction unit 42 to that effect.

[0037] As an example, as shown in Figure 5, the extraction unit 42 includes an analysis unit 70, a selection unit 71, a non-maximum suppression processing unit 72, and a conversion unit 73. The analysis unit 70 inputs the upper tomographic image 15 to the point extraction SS model 32 and outputs a probability distribution map 74 from the point extraction SS model 32 that shows the probability of the presence of the central point CP of the vertebral body. The analysis unit 70 outputs the probability distribution map 74 to the selection unit 71.

[0038] As an example, as shown in Figure 6, the probability distribution map 74 has elements 80 that correspond one-to-one with the pixels 136 of the upper tomographic image 15, and the element values ​​of each element 80 are data in which pairs of probabilities of existence and non-existence of the central point CP of the vertebral body are registered. For example, the element value (1.0, 0) represents that the probability of existence of the central point CP of the vertebral body is 100% and the probability of non-existence is 0%. The probability distribution map 74 is an example of a "feature map" relating to the technology of this disclosure. In addition, the element values ​​of each element 80 of the probability distribution map 74 are an example of a "feature" relating to the technology of this disclosure.

[0039] Returning to Figure 5, the selection unit 71 selects elements 80 in the probability distribution map 74 whose probability of existence of the vertebral center point CP is greater than or equal to a threshold (e.g., 0.9) as candidate vertebral center point CP 120 (see Figure 13). The selection unit 71 generates a point candidate image 75 (see also Figure 13) representing the selected candidate 120 and outputs the generated point candidate image 75 to the non-maximum suppression processing unit 72. The point candidate image 75 is, for example, an image in which the pixel value of the pixel corresponding to candidate 120 is 1 and the pixel values ​​of the other pixels are 0.

[0040] The non-maximum suppression processing unit 72 applies non-maximum suppression processing to each candidate 120 of the candidate point image 75, and as a result generates a point image 76 (see also Figure 13) representing the center point CP of the vertebral body. The point image 76 is, for example, an image in which the pixel value of the pixel corresponding to the center point CP of the vertebral body is 1, and the pixel value of the other pixels is 0. In other words, the point image 76 is an image that identifies the center point CP of the vertebral body.

[0041] The non-maximum suppression processing unit 72 outputs the point image 76 to the conversion unit 73. The conversion unit 73 converts the point image 76 into point position information 50.

[0042] As an example, as shown in Figure 7, the point extraction SS model 32 has a compression unit 90 and an output unit 91. The upper tomographic image 15 is input to the compression unit 90. The upper tomographic image 15 input to the compression unit 90 is, for example, the upper tomographic image 15S of the sagittal section that was the basis for generating the point position display map 51. The compression unit 90 converts the upper tomographic image 15 into a feature map 92. The compression unit 90 passes the feature map 92 to the output unit 91. The output unit 91 outputs a probability distribution map 74 based on the feature map 92.

[0043] The compression unit 90 converts the upper tomographic image 15 into a feature map 92 by performing a convolution operation, as shown in Figure 8 as an example.

[0044] As an example, as shown in Figure 8, the compression unit 90 has a convolutional layer 95. The convolutional layer 95 applies, for example, a 3x3 filter F to target data 97 having multiple elements 96 arranged in two dimensions. It then convolves the element value e of one of the elements 96 of interest 96I with the element values ​​a, b, c, d, f, g, h, and i of eight elements 96S adjacent to the element of interest 96I. The convolutional layer 95 sequentially performs the convolution operation on each element 96 of the target data 97 while shifting the element of interest 96I one element at a time, and outputs the element values ​​of elements 98 of the calculated data 99. This results in calculated data 99 having multiple elements 98 arranged in two dimensions. The target data 97 input to the convolutional layer 95 is, for example, an upper tomographic image 15, or reduced calculated data 99S (see Figure 10), which will be described later.

[0045] If the coefficients of the filter F are r, s, t, u, v, w, x, y, and z, the element value k of element 98I of the calculation data 99, which is the result of the convolution operation on element 96I of interest, can be obtained, for example, by calculating equation (1) below. k=az+by+cx+dw+ev+fu+gt+hs+ir...(1)

[0046] One piece of calculated data 99 is output for each filter F. If multiple types of filters F are applied to a single target data 97, one piece of calculated data 99 is output for each filter F. In other words, as shown in Figure 9 as an example, the number of calculated data 99 generated is equal to the number of filters F applied to the target data 97. Furthermore, since the calculated data 99 has multiple elements 98 arranged in two dimensions, it has a width and a height. The number of calculated data 99 is called the number of channels. Figure 9 illustrates the output of 4-channel calculated data 99, which is generated when 4 filters F are applied to a single target data 97.

[0047] As an example, as shown in Figure 10, the compression unit 90 has a pooling layer 105 in addition to the convolutional layer 95. The pooling layer 105 calculates local statistics of the element values ​​of elements 98 of the calculation data 99 and generates reduced calculation data 99S using the calculated statistics as element values. Here, the pooling layer 105 performs a maximum value pooling process to find the maximum value of element values ​​within a 2x2 block 106 as the local statistics. By shifting the block 106 by one element in the width and height directions while performing the process, the reduced calculation data 99S is reduced to half the size of the original calculation data 99. Figure 10 illustrates the case where the maximum value of element b among a, b, e, and f in block 106A, the maximum value of element b among b, c, f, and g in block 106B, and the maximum value of element h among c, d, g, and h in block 106C are all the maximum values. Alternatively, mean pooling can be used to calculate the mean value as the local statistic, rather than the maximum value.

[0048] The compression unit 90 outputs the final calculated data 99 by repeatedly performing convolution processing by the convolutional layer 95 and pooling processing by the pooling layer 105. This final calculated data 99 is the feature map 92. Although not shown in the diagram, the compression unit 90 also performs skip layer processing, etc., to pass the calculated data 99 to the output unit 91.

[0049] The output unit 91 performs an upsampling process to enlarge the feature map 92, thereby creating an enlarged feature map. The output unit 91 also performs a convolution process simultaneously with the upsampling process. Furthermore, the output unit 91 performs a merge process to combine the enlarged feature map with the computational data 99 received from the compression unit 90 in the skip layer process. The output unit 91 performs another convolution process after the merge process. After these various processes, the output unit 91 outputs a probability distribution map 74 from the feature map 92.

[0050] Thus, the SS model 32 for point extraction is constructed by a CNN. Examples of the CNN include U-Net, ResNet (Residual Network), and the like.

[0051] As an example, as shown in FIG. 11, the output unit 91 includes a decoder unit 110 and a probability distribution map generation unit 111. The decoder unit 110 performs upsampling processing, convolution processing, merge processing, and the like on the feature map 92 as described above, and generates a final feature map 112. The final feature map 112, also called logits, has elements corresponding one-to-one with the pixels 136 of the upper tomographic image 15. Each element of the final feature map 112 has an element value related to the center point CP of the vertebral body to be extracted. For example, the element value of an element where the center point CP of the vertebral body is considered to be is higher than the element values of other elements. The decoder unit 110 outputs the final feature map 112 to the probability distribution map generation unit 111.

[0052] The probability distribution map generation unit 111 generates a probability distribution map 74 from the final feature map 112 using a well-known activation function.

[0053] For example, consider a case where, in a certain element of the final feature map 112, the element value assuming it is the center point CP of the vertebral body is 2, and the element value assuming it is not the center point CP of the vertebral body is 1.5. In this case, the probability distribution map generation unit 111 applies, for example, the softmax function, and calculates e 2 / (e 2 +e 1.5 ) and e 1.5 / (e 2 +e 1.5 ). Then, the probability that the center point CP of the vertebral body exists in the said element, that is, the existence probability, is derived as 0.6(≒e 2 / (e 2 +e 1.5 )), and the probability that the center point CP of the vertebral body does not exist in the said element (hereinafter referred to as non-existence probability) is derived as 0.38(≒e 1.5 / (e 2 +e 1.5 )). Instead of the softmax function, a sigmoid function may be used.

[0054] As an example, as shown in Figure 12, the point extraction SS model 32 is trained in the training phase by being given training data (also called training data) 115. The training data 115 is a set of a training upper tomographic image 15L and an annotation image 116 corresponding to the training upper tomographic image 15L. The annotation image 116 is an image in which the center point CP of the vertebral body of each vertebra VB shown in the training upper tomographic image 15L is annotated. The annotation image 116 is an image in which the pixel value of pixel 117A corresponding to the center point CP of the vertebral body is set to 1, and the pixel values ​​of the other pixels 117B are set to 0. Note that pixel 117A is an example of a "first pixel" related to the technology of this disclosure, and the pixel value of pixel 117A, which is 1, is an example of a "first pixel value" related to the technology of this disclosure. Furthermore, pixel 117B is an example of a "second pixel" relating to the technology of this disclosure, and the pixel value of pixel 117B being 0 is an example of a "second pixel value" relating to the technology of this disclosure.

[0055] During the learning phase, the point extraction SS model 32 receives the training upper tomographic image 15L as input. The point extraction SS model 32 outputs a training probability distribution map 74L based on the training upper tomographic image 15L. Based on this training probability distribution map 74L and the annotation image 116, the point extraction SS model 32 performs a loss calculation. Then, according to the result of the loss calculation, various coefficients of the point extraction SS model 32 (such as the coefficient of filter F) are updated, and the point extraction SS model 32 is updated according to the update settings.

[0056] In the loss calculation of the point extraction SS model 32, a weighted cross-entropy function is used. The cross-entropy function takes a relatively low value when the probability of existence of the vertebral center point CP among the element values ​​80 of the training probability distribution map 74L and the pixel value of the annotation image 116 are relatively close. In other words, in this case, the loss is estimated to be small. Conversely, the cross-entropy function takes a relatively high value when the probability of existence of the vertebral center point CP among the element values ​​80 of the training probability distribution map 74L and the pixel value of the annotation image 116 are relatively far apart. In other words, in this case, the loss is estimated to be large.

[0057] The weights of the cross-entropy function are set as follows: for example, 10 is assigned to pixel 117A corresponding to the center point CP of the annotated vertebra in the annotation image 116, and 1 is assigned to the other pixels 117B.

[0058] In the training phase of the point extraction SS model 32, the above series of processes—inputting the training upper tomographic image 15L into the point extraction SS model 32, outputting the training probability distribution map 74L from the point extraction SS model 32, loss calculation, update settings, and updating the point extraction SS model 32—are repeated while the training data 115 is exchanged. The repetition of the above series of processes ends when the prediction accuracy of the training probability distribution map 74L for the annotation image 116 reaches a predetermined set level. The point extraction SS model 32, whose prediction accuracy has reached the set level, is stored in the storage 20 and used in the extraction unit 42. Alternatively, training may be terminated after the above series of processes has been repeated a set number of times, regardless of the prediction accuracy of the training probability distribution map 74L for the annotation image 116.

[0059] Figure 13 shows an example of non-maximal suppression processing by the non-maximal suppression processing unit 72. The candidate point image 75 is simply selected as candidate 120 if the probability of existence of elements 80 in the probability distribution map 74 is above a threshold. Therefore, not all candidate 120 are necessarily the true center point CP of the vertebral body. By applying non-maximal suppression processing, the true center point CP of the vertebral body is narrowed down from among the multiple candidate 120.

[0060] The non-maximum suppression processing unit 72 first assigns a rectangular frame 121 to each candidate 120 in the point candidate image 75. The rectangular frame 121 has a preset size corresponding to the vertebral VB, for example, a size slightly larger than one vertebral VB. The center of the rectangular frame 121 coincides with the candidate 120.

[0061] Next, the non-maximum suppression processing unit 72 calculates the IoU (Intersection over Union) of the rectangular frame 121 assigned to each candidate 120. IoU is the value obtained by dividing the overlapping area of ​​the two rectangular frames 121 by the area of ​​union formed by integrating the two rectangular frames 121. For two rectangular frames 121 whose IoU is greater than or equal to a threshold (e.g., 0.3), the non-maximum suppression processing unit 72 leaves one representative rectangular frame 121 and assigns the other rectangular frame 121 to candidate 120. each Delete. This merges two rectangular frames 121 with IoU above the threshold into one rectangular frame 121. By deleting adjacent rectangular frames 121 and overlapping rectangular frames 121 and candidate 120 with IoU above the threshold in this way, a point image 76 representing the center point CP of the vertebral body is finally obtained.

[0062] As an example, as shown in Figure 14, the conversion unit 73 generates point position information 50 based on the point image 76. The point position information 50 is the XYZ coordinates of the position of the center point CP of the vertebral body in the point image 76. The X axis is parallel to the left-right direction, the Y axis is parallel to the front-back direction, and the Z axis is parallel to the up-down direction. In this example, since the upper tomographic image 15S of the sagittal section is used, the value of the X coordinate among the XYZ coordinates of each center point CP is the same for each center point CP. The values ​​of the Y coordinate and Z coordinate differ depending on each center point CP. In the point position information 50, the center points CP of each vertebral body are organized by assigning numbers (No.) in ascending order of the Z coordinate.

[0063] The point location display map 51 has elements 137 (see Figure 16) that correspond one-to-one with pixels 136 of the upper tomographic image 15. The element value of element 137 corresponding to pixel 136 of the vertebral body's center point CP is set to 1 or 2, and the element value of element 137 corresponding to pixels 136 other than the vertebral body's center point CP is set to 0. In other words, the point location display map 51 is data that represents the position of the vertebral body's center point CP with an element value of 1 or 2. Note that in Figure 14, vertebral bones VB, etc., are shown with dashed lines to aid understanding, but vertebral bones VB, etc., are not actually shown in the point location display map 51.

[0064] The point position display map generation unit 43 assigns label A to two adjacent vertebrae VB by setting the element value of element 137 corresponding to the center point CP of one vertebrae VB to 1, and label B to element 137 corresponding to the center point CP of the other vertebrae VB by setting the element value of element 137 to 2. For example, label A is assigned to element 137 corresponding to the center point CP of vertebrae No. 1, and label B is assigned to element 137 corresponding to the center point CP of vertebrae No. 2, and label B is assigned to element 137, and label A is assigned to element 137, and label B the element value of element 137, corresponding to the center point CP of vertebrae No. 8, respectively. The point position display map generation unit 43, by thus assigning labels A and B, assigns label A to the element 137 corresponding to the center point CP of vertebral bodies No. 1, 3, 5, 7, and 9, and labels B to the element 137 corresponding to the center point CP of vertebral bodies No. 2, 4, 6, and 8. In other words, labels A and B are assigned alternately to the center point CP of each vertebral body VB.

[0065] As an example, as shown in Figure 15, the object recognition SS model 33, like the point extraction SS model 32, has a compression unit 130 and an output unit 131. The compression unit 130 receives an upper tomographic image 15 and a point location display map 51 as input. The upper tomographic image 15 input to the compression unit 130 is, for example, the upper tomographic image 15S of the sagittal section that was the basis for generating the point location display map 51. The compression unit 130 converts the upper tomographic image 15 and the point location display map 51 into a feature map 132. The compression unit 130 passes the feature map 132 to the output unit 131. The output unit 131 outputs an output image 52 based on the feature map 132.

[0066] The compression unit 130 combines the upper tomographic image 15 and the point position display map 51 in the channel direction by performing a convolution operation, as shown in Figure 16 as an example.

[0067] The compression unit 130 has a convolutional layer 135 into which the upper tomography image 15 and the point location display map 51 are input. The convolutional layer 135 applies, for example, a 3x3 filter F1 to the upper tomography image 15 which has multiple pixels 136 arranged in two dimensions. The convolutional layer 135 also applies, for example, a 3x3 filter F2 to the point location display map 51 which has multiple elements 137 arranged in two dimensions. Then, the pixel value e1 of one of the pixels 136, the pixel of interest 136I, and the pixel values ​​a1, b1, c1, d1, f1, g1, h1, and i1 of the eight pixels 136S adjacent to the pixel of interest 136I, as well as the element value e2 of one of the elements 137, the element of interest 137I corresponding to the pixel of interest 136I, and the element values ​​a2, b2, c2, d2, f2, g2, h2, and i2 of the eight elements 137S adjacent to the element of interest 137I, are convolved. The convolution layer 135 sequentially performs the convolution operation while shifting the pixel of interest 136I and the element of interest 137I one by one, and outputs the element values ​​of element 139 of the calculation data 138. This results in calculation data 138 having multiple elements 139 arranged in two dimensions. In this way, the upper tomographic image 15 and the point position display map 51 are combined in the channel direction.

[0068] Let the coefficients of filter F1 be r1, s1, t1, u1, v1, w1, x1, y1, and z1. Also, let the coefficients of filter F2 be r2, s2, t2, u2, v2, w2, x2, y2, and z2. In this case, the element value k of element 139I of the calculation data 138, which is the result of the convolution operation on the pixel of interest 136I and the element of interest 137I, can be obtained, for example, by calculating equation (2) below. k=a1z1+b1y1+c1x1+d1w1+e1v1+f1u1+g1t1+h1s1+i1r1+a2z2+b2y2+c2x2+d2w2+e2v2+f2u2+g2t2+h2s2+i2r2...(2)

[0069] The compression unit 130 has multiple convolutional layers similar to the convolutional layer 95 shown in Figure 8, in addition to the convolutional layer 135, and performs convolution multiple times. The compression unit 130 also has multiple pooling layers similar to the pooling layer 105 shown in Figure 10, and performs pooling multiple times. Furthermore, the compression unit 130 also performs a skip layer process to pass the calculated data from the convolution process to the output unit 131. By repeating the convolution process, pooling process, and skip layer process multiple times, the compression unit 130 outputs the final calculated data 138. This final calculated data 138 is the feature map 132.

[0070] The output unit 131 performs upsampling, convolution, and merging on the feature map 132, similar to the output unit 91 of the point extraction SS model 32. After these various processes, the output unit 131 outputs the output image 52 from the feature map 132. In this way, the object recognition SS model 33 is constructed using a CNN, similar to the point extraction SS model 32.

[0071] As an example, as shown in Figure 17, the output image 52 is an image in which a class is labeled for each vertebral body VB. More specifically, the output image 52 is an image in which the vertebral bodies VBs containing the central points CP of vertebral bodies No. 1, 3, 5, 7, and 9 are identified as Class A, corresponding to label A, and the vertebral bodies VBs containing the central points CP of vertebral bodies No. 2, 4, 6, and 8 are identified as Class B, corresponding to label B.

[0072] As an example, as shown in Figure 18, the object recognition SS model 33 is trained in the training phase by being given training data 150. The training data 150 is a set consisting of a training upper tomographic image 15L, a training point position display map 51L corresponding to the training upper tomographic image 15L, and an annotation image 151 corresponding to the training upper tomographic image 15L and the training point position display map 51L. The training point position display map 51L is created by alternately labeling elements 137 corresponding to the center point CP of the vertebral body of vertebra VB shown in the training upper tomographic image 15L with labels A and B. The annotation image 151 is an image in which each vertebra VB shown in the training upper tomographic image 15L is labeled with a class corresponding to the label applied in the training point position display map 51L.

[0073] During the learning phase, the object recognition SS model 33 receives a training upper tomographic image 15L and a training point location display map 51L as input. The object recognition SS model 33 outputs a training output image 52L based on the training upper tomographic image 15L and the training point location display map 51L. Based on this training output image 52L and the annotation image 151, the object recognition SS model 33 performs a loss calculation. Then, according to the result of the loss calculation, various coefficients of the object recognition SS model 33 (such as the coefficients of filters F1 and F2) are updated, and the object recognition SS model 33 is updated according to the update settings.

[0074] In the training phase of the object recognition SS model 33, the above series of processes—input of the training upper tomographic image 15L and the training point position display map 51L into the object recognition SS model 33, output of the training output image 52L from the object recognition SS model 33, loss calculation, update settings, and updating the object recognition SS model 33—are repeated while the training data 150 is exchanged. The repetition of the above series of processes ends when the prediction accuracy of the training output image 52L for the annotation image 151 reaches a predetermined set level. The object recognition SS model 33, whose prediction accuracy has reached the set level, is stored in the storage 20 and used in the object recognition unit 44. Note that training may be terminated after the above series of processes has been repeated a set number of times, regardless of the prediction accuracy of the training output image 52L for the annotation image 151.

[0075] As an example, as shown in Figure 19, the assigned result 53 is the anatomical name of each vertebra VB, such as Th10 (10th thoracic vertebra), L1 (1st lumbar vertebra), and S1 (sacrum).

[0076] Figure 20 shows an example of the second screen 155 displaying the assigned result 53. The display control unit 46 transitions the screen from the first screen 60 shown in Figure 4 to the second screen 155. The second screen 155 displays the upper tomographic image 1 5 The assigned result 53 is displayed to the side. Also, the second screen 155 displays the findings input field 62 and the OK button 64, just like the first screen 60 in Figure 4. The doctor enters the findings in the findings input field 62 while referring to the assigned result 53, and then selects the OK button 64 by moving the cursor 66 over it. As a result, the instruction receiving unit 41 receives the finding storage instruction, just as in the case of Figure 4. The RW control unit 40 associates the upper tomographic image 15 with the findings entered in the findings input field 62 and stores them in the storage 20.

[0077] Next, the operation of the above configuration will be explained with reference to the flowchart in Figure 21. First, when the operating program 30 is started in the diagnostic support device 12, as shown in Figure 3, the CPU 22 of the diagnostic support device 12 functions as the RW control unit 40, instruction reception unit 41, extraction unit 42, point position display map generation unit 43, object identification unit 44, anatomical naming unit 45, and display control unit 46.

[0078] The RW control unit 40 reads the upper tomographic image 15 of patient P, who is being diagnosed with spinal SP, from the storage 20 (step ST100). The upper tomographic image 15 is output from the RW control unit 40 to the display control unit 46. Then, under the control of the display control unit 46, the first screen 60 shown in Figure 4 is displayed on the display 17 (step ST110).

[0079] On the first screen 60, if the physician selects the analysis button 65, the instruction to analyze the upper tomographic image 15 is received by the instruction reception unit 41 (step ST120). As a result, as shown in Figure 5, the upper tomographic image 15 is input to the point extraction SS model 32 in the analysis unit 70 of the extraction unit 42, and a probability distribution map 74 showing the probability of the presence of the central point CP of the vertebral body is output from the point extraction SS model 32 (step ST130). The probability distribution map 74 is output from the analysis unit 70 to the selection unit 71.

[0080] As shown in Figure 13, the selection unit 71 selects elements 80 in the probability distribution map 74 whose probability of existence of the vertebral center point CP is greater than or equal to a threshold as candidate 120 for the vertebral center point CP (step ST140). Then, the non-maximal suppression processing unit 72 assigns a rectangular frame 121 to the candidate 120 and applies non-maximal suppression processing to the rectangular frame 121. This generates a point image 76 representing the vertebral center point CP (step ST150). The point image 76 is output from the non-maximal suppression processing unit 72 to the conversion unit 73.

[0081] As shown in Figure 14, the conversion unit 73 generates point position information 50 representing the XYZ coordinates of the central point CP of the vertebral body in the point image 76, based on the point image 76 (step ST160). The point position information 50 is output from the conversion unit 73 to the point position display map generation unit 43.

[0082] As shown in Figure 14, the point location display map generation unit 43 generates a point location display map 51 based on the point location information 50 (step ST170). The point location display map 51 is output from the point location display map generation unit 43 to the object identification unit 44.

[0083] The object identification unit 44 receives the upper tomographic image 15 and the object identification SS model 33 from the RW control unit 40. In the object identification unit 44, as shown in Figure 15, the upper tomographic image 15 and the point position display map 51 are input to the object identification SS model 33. At this time, as shown in Figure 16, the upper tomographic image 15 and the point position display map 51 are combined in the channel direction. Then, the object identification SS model 33 outputs the output image 52 (step ST180). The output image 52 is output from the object identification unit 44 to the anatomical naming unit 45.

[0084] In the anatomical naming unit 45, an anatomical name is assigned to each vertebra VB identified in the output image 52, as shown in Figure 19 (step ST190). The assignment result 53 is output from the anatomical naming unit 45 to the display control unit 46.

[0085] Under the control of the display control unit 46, the second screen 155 shown in Figure 20 is displayed on the display 17 (step ST200). The physician enters findings in the findings input field 62 while referring to the assigned result 53, and then selects the OK button 64 by moving the cursor 66 to it. This allows the findings storage instruction to be received by the instruction reception unit 41 (step ST210). Then, under the control of the RW control unit 40, the upper tomographic image 15 and the findings entered in the findings input field 62 are associated and stored in the storage 20 (step ST220).

[0086] As explained above, the diagnostic support device 12 uses a point extraction SS model 32. As shown in Figure 12, the point extraction SS model 32 is trained using an annotation image 116 in which the pixel value of pixel 117A corresponding to the center point CP of each vertebra VB is set to 1, and the pixel value of the other pixels 117B is set to 0. The point extraction SS model 32 is trained by assigning a larger weight to the pixel 117A corresponding to the center point CP of the vertebra than to the other pixels 117B and calculating the loss. As shown in Figure 5, the extraction unit 42 inputs the upper tomographic image 15 to the point extraction SS model 32 and outputs a probability distribution map 74 from the point extraction SS model 32 that shows the probability of the presence of the center point CP of the vertebra in the upper tomographic image 15. Then, as shown in Figure 13, a point image 76 is generated based on the probability distribution map 74 and the center point CP of the vertebra is identified. Since the center point CP of the vertebra is very small, it would be difficult to learn without any countermeasures, as it would be buried. However, by assigning a larger weight to pixel 117A, which corresponds to the center point CP of the vertebral body, than to the other pixels 117B, it is possible to focus the learning on the center point CP of the vertebral body. Therefore, it becomes possible to accurately identify the center point CP of the vertebral body.

[0087] As shown in Figure 13, the selection unit 71 selects elements 80 from the elements 80 of the probability distribution map 74 that have a probability of existence above a predetermined threshold as candidates 120 for the center point CP of the vertebral body. The non-maximal suppression processing unit 72 assigns a rectangular frame 121 of a predetermined size to the selected candidates 120. Then, it applies non-maximal suppression processing to the rectangular frame 121 and identifies the center point CP of the vertebral body based on the results of the non-maximal suppression processing. In this way, multiple discrete points that are relatively close to each other, such as the center point CP of a vertebral body, can be identified with high accuracy.

[0088] In the medical field, there is a very high demand for accurate identification of bodily structures to aid in accurate diagnosis. Therefore, this example, which uses the upper tomographic image 15, a medical image showing the inside of patient P's body, as the target image for analysis, and the vertebra VB, a bodily structure, as the target object, can be said to be a form that matches this demand.

[0089] The upper tomographic images 15 input to the compression units 90 and 130 are not limited to the upper tomographic images 15S of the sagittal section that were used to generate the point position display map 51. In addition to the upper tomographic images 15S of the sagittal section that were used to generate the point position display map 51, several upper tomographic images 15S of sagittal sections before and after the upper tomographic image 15S of the sagittal section that were used to generate the point position display map 51 may also be input to the compression units 90 and 130. Alternatively, the identification result of vertebral VB for one upper tomographic image 15S of sagittal section may be applied to several upper tomographic images 15S of sagittal sections before and after it.

[0090] The annotated image 116 is not limited to an image in which a single pixel 117A indicating the central point CP of the vertebral body is annotated. It may also be an image in which a circular region consisting of several to tens of pixels centered on the central point CP of the vertebral body is annotated. Furthermore, the points to be extracted are not limited to the central point CP of the vertebral body. They may be the tips of the spinous processes of the vertebral arch or the center of the vertebral foramen.

[0091] Note that discrete points that are relatively close to each other are not limited to the central point CP of the vertebral body. They could also be the central points of the finger bones, etc.

[0092] In the example above, the element value of label A is set to 1 and the element value of label B is set to 2, but this is not the only option. The element values ​​of label A and label B just need to be different from each other. For example, the element value of label A could be set to 1 and the element value of label B could be set to -1.

[0093] The types of labels are not limited to just two: labels A and B. Three or more labels may be assigned. For example, vertebral VBs No. 1, 4, and 7 may be assigned label A with an element value of 1, vertebral VBs No. 2, 5, and 8 may be assigned label B with an element value of 2, and vertebral VBs No. 3, 6, and 9 may be assigned label C with an element value of 3. In this case, the output image 52 will be an image in which vertebral VBs No. 1, 4, and 7 are identified as class A corresponding to label A, vertebral VBs No. 2, 5, and 8 are identified as class B corresponding to label B, and vertebral VBs No. 3, 6, and 9 are identified as class C corresponding to label C.

[0094] In the above example, we illustrated a method of identifying vertebral vBs in order to assign an anatomical name to each vertebral vB, but this is not limited to this. For example, vertebral vBs may be identified as a preprocessing step for CAD (Computer-Aided Diagnosis) to extract candidate lesions such as fractures and bone metastases from cancer.

[0095] [Second Embodiment] In the second embodiment shown in Figures 22 to 25, instead of the points within the vertebra VB in the first embodiment, points within the left and right eyeballs EB are identified as discrete points.

[0096] As an example, as shown in Figure 22, the second embodiment uses a tomographic image 160 of the head showing the left and right eyeballs (EBs). The tomographic image 160 is an example of an "image to be analyzed" and a "medical image" related to the technology of this disclosure. The eyeballs (EBs) are an example of an "object" and a "structure" related to the technology of this disclosure.

[0097] The extraction unit 165 of this embodiment includes an analysis unit 166 and a centroid calculation unit 167. The analysis unit 166 inputs the head tomography image 160 to a point extraction SS model 168 and outputs an output image 169 from the point extraction SS model 168. The output image 169 is an image in which regions that appear to be the center points of the left and right eyeballs EB in the head tomography image 160 are labeled as classes. The analysis unit 166 outputs the output image 169 to the centroid calculation unit 167. The centroid calculation unit 167 generates point position information 170 indicating the center points of the left and right eyeballs EB based on the output image 169. The center points of the left and right eyeballs EB are an example of "a plurality of discrete points" related to the technology of this disclosure.

[0098] As an example, as shown in Figure 23, the point extraction SS model 168, like the point extraction SS model 32, has a compression unit 175 and an output unit 176, and is constructed using a CNN. The compression unit 175 receives a head tomography image 160 as input. The compression unit 175 converts the head tomography image 160 into a feature map 177. The compression unit 175 passes the feature map 177 to the output unit 176. The output unit 176 outputs an output image 169 based on the feature map 177. The point extraction SS model 168, like the point extraction SS model 32 in the first embodiment described above, is learned by assigning a larger weight to the pixels corresponding to the center points of the left and right eyeballs EB than to the other pixels and calculating the loss.

[0099] As an example, as shown in Figure 24, the output unit 176, similar to the output unit 91 of the point extraction SS model 32, has a decoder unit 180 and a probability distribution map generation unit 181. The output unit 176 also has a labeling unit 182. The decoder unit 180 performs upsampling, convolution, and merging on the feature map 177 to generate the final feature map 183. The final feature map 183 has elements that correspond one-to-one with the pixels of the head tomography image 160. Each element of the final feature map 183 has an element value related to the center points of the left and right eyeballs EB that are to be extracted. For example, the element value of the element that is thought to contain the center points of the left and right eyeballs EB is higher than the element values ​​of other elements. The decoder unit 180 outputs the final feature map 183 to the probability distribution map generation unit 181.

[0100] The probability distribution map generation unit 181 generates a probability distribution map 184 from the final feature map 183. The probability distribution map 184 shows the probability of existence of the center points EB of the left and right eyeballs. The probability distribution map generation unit 181 outputs the probability distribution map 184 to the labeling unit 182. The probability distribution map 184 is an example of a "feature map" related to the technology of this disclosure.

[0101] The labeling unit 182 labels each element of the probability distribution map 184 with either a class indicating that it is the center point of the eyeball EB, or a class indicating that it is not the center point of the eyeball EB. The labeling unit 182 labels elements where the probability of existence is greater than the probability of non-existence (probability of existence > probability of non-existence) with the class indicating that it is the center point of the eyeball EB. On the other hand, the labeling unit 182 labels elements where the probability of existence is less than or equal to the probability of non-existence (probability of existence ≤ probability of non-existence) with the class indicating that it is not the center point of the eyeball EB. As a result, an output image 169 is obtained in which the regions that appear to be the center points of the left and right eyeball EBs are labeled as classes.

[0102] As an example, as shown in Figure 25, the centroid calculation unit 167 calculates the centroid CG of each of the two regions labeled as the center points of the left and right eyeballs EB in the output image 169. The centroid calculation unit 167 generates point position information 170 in which the coordinates of the calculated centroid CG are registered as the coordinates of the center points of the left and right eyeballs EB.

[0103] Thus, in the second embodiment, the extraction unit 165 generates an output image 169 in which each pixel is labeled with a class corresponding to the probability of existence of the probability distribution map 184. Then, based on the output image 169, the center points of the left and right eyeballs EB are identified. Therefore, multiple discrete points that are relatively far apart from each other, such as the center points of the left and right eyeballs EB, can be identified with high accuracy.

[0104] Note that discrete points that are relatively far apart from each other are not limited to the center points of the eyeballs (EB) of the left and right eyes. They could also be the center points of the right and left hippocampi of the brain, for example.

[0105] [Third Embodiment] In the first and second embodiments described above, discrete points corresponding to multiple objects were identified, but the invention is not limited to this. As shown in the third embodiment in Figures 26 to 28, a single point corresponding to an object may be identified.

[0106] As an example, as shown in Figure 26, the third embodiment uses a chest tomography image 190 showing the aortic valve AV. The chest tomography image 190 is an example of an "image to be analyzed" and a "medical image" related to the technology of this disclosure. The aortic valve AV is an example of an "object" and a "structure" related to the technology of this disclosure.

[0107] The extraction unit 195 of this embodiment includes an analysis unit 196 and a centroid calculation unit 197. The analysis unit 196 inputs the chest tomography image 190 to a point extraction SS model 198 and outputs a probability distribution map 199 from the point extraction SS model 198. The probability distribution map 199 shows the probability of existence of the center point of the aortic valve AV. The analysis unit 196 outputs the probability distribution map 199 to the centroid calculation unit 197. The centroid calculation unit 197 generates point position information 200 indicating the center point of the aortic valve AV based on the probability distribution map 199. The point extraction SS model 198 is learned by assigning a larger weight to the pixel corresponding to the center point of the aortic valve AV than to other pixels and calculating the loss, similar to the point extraction SS model 32 of the first embodiment described above. The center point of the aortic valve AV is an example of "one point" related to the technology of this disclosure. Furthermore, probability distribution map 199 is an example of a "feature map" relating to the technology of this disclosure.

[0108] As an example, as shown in Figure 27, the centroid calculation unit 197 calculates the centroid CG of the probability of existence of the center point of the aortic valve AV of the element values ​​of the probability distribution map 199. The centroid CG is expressed by the following equation (3), where r is the coordinate vector of each element of the probability distribution map 199 and ρ(r) is the probability of existence of the center point of the aortic valve AV of the element values ​​of the probability distribution map 199. CG=(Σρ(r)·r) / Σρ(r)···(3) In other words, the centroid CG is the value obtained by dividing the sum of the products of the probability of existence ρ(r) of the center point of the aortic valve AV and the coordinate vector r of each element of the probability distribution map 199 by the sum of the probabilities of existence ρ(r) of the center point of the aortic valve AV. The centroid calculation unit 197 generates point position information 200 in which the coordinates of the calculated centroid CG are registered as the coordinates of the center point of the aortic valve AV.

[0109] Thus, in the third embodiment, the extraction unit 195 identifies the centroid CG of the probability distribution map 199, which is based on the probability of the existence of the center point of the aortic valve AV, as a single point. Therefore, a single point, such as the center point of the aortic valve AV, can be identified with high accuracy.

[0110] As an example, as shown in Figure 28, the center point of the aortic valve AV may be identified based on the final feature map 214 rather than the probability distribution map 199.

[0111] In Figure 28, the extraction unit 210 includes an analysis unit 211 and a selection unit 212. The analysis unit 211 inputs the chest tomography image 190 into a point extraction SS model 213 and outputs a final feature map 214 from the point extraction SS model 213. The final feature map 214 has elements that correspond one-to-one with the pixels of the chest tomography image 190. Each element of the final feature map 214 has an element value related to the center point of the aortic valve AV that is the target of extraction. For example, the element value of an element that is thought to contain the aortic valve AV is higher than the element values ​​of other elements. The final feature map 214 is an example of a "feature map" related to the technology of this disclosure. The analysis unit 211 outputs the final feature map 214 to the selection unit 212. Furthermore, the point extraction SS model 213, like the point extraction SS model 198, is trained by assigning a larger weight to the pixel corresponding to the center point of the aortic valve AV than to the other pixels and calculating the loss.

[0112] The selection unit 212 selects the element with the largest element value in the final feature map 214 as the centroid of the final feature map 214. The selection unit 212 generates point position information 215, which registers the coordinates of the element with the largest element value in the final feature map 214 as the coordinates of the center point of the aortic valve AV.

[0113] In this way, the extraction unit 210 identifies the centroid (the element with the largest element value) of the final feature map 214 as a single point. This method also allows for the accurate identification of a single point, such as the center point of the aortic valve AV.

[0114] Note that the single point is not limited to the center point of the aortic valve (AV). It could also be the center point of the pulmonary valve, or the center point of the mitral valve, etc.

[0115] [Fourth Embodiment] In the embodiments described above, points were identified, but the invention is not limited to this. As in the fourth embodiment shown in Figures 29 to 35, lines corresponding to objects having a linear structure may also be identified.

[0116] As an example, as shown in Figure 29, the fourth embodiment deals with a cardiac tomographic image 220 showing the aortic AO of the heart. The cardiac tomographic image 220 is an example of an "image to be analyzed" and a "medical image" related to the technology of this disclosure. The aortic AO is an example of an "object" and a "structure" related to the technology of this disclosure.

[0117] The extraction unit 225 of this embodiment includes an analysis unit 226 and a thinning processing unit 227. The analysis unit 226 inputs the cardiac tomography image 220 to a line extraction SS model 228 and outputs an output image 229 from the line extraction SS model 228. The output image 229 is an image in which the region that appears to be the centerline of the aorta AO in the cardiac tomography image 220 is labeled as a class. The analysis unit 226 outputs the output image 229 to the thinning processing unit 227. The thinning processing unit 227 generates line position information 230 indicating the centerline of the aorta AO based on the output image 229. The centerline of the aorta AO is an example of a "line" related to the technology of this disclosure.

[0118] As an example, as shown in Figure 30, the line extraction SS model 228, like the point extraction SS model 32, has a compression unit 235 and an output unit 236, and is constructed using a CNN. The compression unit 235 receives a cardiac tomography image 220 as input. The compression unit 235 converts the cardiac tomography image 220 into a feature map 237. The compression unit 235 passes the feature map 237 to the output unit 236. The output unit 236 outputs an output image 229 based on the feature map 237. The line extraction SS model 228, like the point extraction SS model 32 in the first embodiment described above, is learned by assigning a larger weight to the pixels corresponding to the centerline of the aorta AO than to the other pixels and calculating the loss.

[0119] As an example, as shown in Figure 31, the output unit 236, similar to the output unit 176 of the point extraction SS model 168 in the second embodiment, includes a decoder unit 240, a probability distribution map generation unit 241, and a labeling unit 242. The decoder unit 240 performs upsampling, convolution, and merging on the feature map 237 to generate the final feature map 243. The final feature map 243 has elements that correspond one-to-one with the pixels of the cardiac tomography image 220. Each element of the final feature map 243 has an element value related to the centerline of the aorta AO, which is the target of extraction. For example, the element value of an element that is thought to contain the centerline of the aorta AO is higher than the element values ​​of other elements. The decoder unit 240 outputs the final feature map 243 to the probability distribution map generation unit 241.

[0120] The probability distribution map generation unit 241 generates a probability distribution map 244 from the final feature map 243. The probability distribution map 244 shows the probability of the existence of the centerline of the aorta AO. The probability distribution map generation unit 241 outputs the probability distribution map 244 to the labeling unit 242. Note that the probability distribution map 244 is an example of a "feature map" related to the technology of this disclosure.

[0121] The labeling unit 242 assigns a class to each element of the probability distribution map 244, indicating that it is the centerline of the aorta AO, and the center of the aorta AO. line The labeling unit 242 labels elements where the probability of existence is greater than the probability of non-existence (probability of existence > probability of non-existence) with a class indicating that it is the centerline of the aorta (AO). On the other hand, the labeling unit 242 labels elements where the probability of existence is less than or equal to the probability of non-existence (probability of existence ≤ probability of non-existence) with a class indicating that it is not the centerline of the aorta (AO). As a result, an output image 229 is obtained in which the region that is thought to be the centerline of the aorta (AO) is labeled with a class.

[0122] As an example, as shown in Figure 32, the thinning processing unit 227 performs a thinning process on the region that appears to be the centerline of the aorta AO, which is labeled as a class in the output image 229, and converts the region that appears to be the centerline of the aorta AO into a thin line TL. A thin line TL is composed of a series of pixels. The thinning processing unit 227 generates line position information 230, which registers the coordinates of each pixel that makes up the thin line TL.

[0123] Thus, in the fourth embodiment, the extraction unit 225 generates an output image 229 in which each pixel is labeled with a class corresponding to the probability of existence of the probability distribution map 244. Then, the output image 229 is subjected to a thinning process, and the centerline of the aorta AO is identified based on the result of the thinning process. Therefore, lines such as the centerline of the aorta AO can be identified with high accuracy.

[0124] As an example, the extraction unit 250 shown in Figure 33 may be used to identify lines such as the centerline of the aorta AO.

[0125] In Figure 33, the extraction unit 250, like the extraction unit 42 in the first embodiment described above, includes an analysis unit 251, a selection unit 252, a non-maximum suppression processing unit 253, and a conversion unit 254. The analysis unit 251 inputs the cardiac tomography image 220 to the line extraction SS model 255 and outputs a probability distribution map 256 from the line extraction SS model 255 that shows the probability of the presence of the centerline of the aorta AO. The analysis unit 251 outputs the probability distribution map 256 to the selection unit 252. The probability distribution map 256 is an example of a "feature map" related to the technology of this disclosure. The line extraction SS model 255, like the line extraction SS model 228, is learned by assigning a larger weight to the pixels corresponding to the centerline of the aorta AO than to other pixels and calculating the loss.

[0126] The selection unit 252 selects elements in the probability distribution map 256 whose probability of existence of the aortic centerline is greater than or equal to a threshold (e.g., 0.9) as candidate 260 for the aortic centerline (see Figure 34). The selection unit 252 generates a line candidate image 257 (see also Figure 34) representing the selected candidate 260 and outputs the generated line candidate image 257 to the non-maximum suppression processing unit 253. The line candidate image 257 is, for example, an image in which the pixel value of the pixel corresponding to candidate 260 is 1 and the pixel value of the other pixels is 0.

[0127] The non-maximal suppression processing unit 253 applies non-maximal suppression processing to each candidate 260 of the line candidate image 257, and as a result generates a line image 258 (see also Figure 35) that represents the centerline of the aorta AO. The line image 258 is an image in which, for example, the pixel value of the pixel corresponding to the centerline of the aorta AO is 1, and the pixel value of the other pixels is 0. In other words, the line image 258 is an image that identifies the centerline of the aorta AO.

[0128] The non-maximum suppression processing unit 253 outputs the line image 258 to the conversion unit 254. The conversion unit 254 converts the line image 258 into line position information 259.

[0129] Figures 34 and 35 show an example of non-maximal suppression processing by the non-maximal suppression processing unit 253. The candidate line image 257 is simply selected as candidate 260 if the probability of existence in the probability distribution map 256 is above a threshold. Therefore, not all candidate 260 are necessarily the true centerline of the aorta (AO). By applying non-maximal suppression processing, the true centerline of the aorta (AO) is narrowed down from among multiple candidate 260.

[0130] As shown in Figure 34, the non-maximum suppression processing unit 253 first assigns a rectangular frame 261 to each candidate 260 of the line candidate image 257. The rectangular frame 261 has a preset size corresponding to the aorta AO, for example, a size slightly larger than the width of the aorta AO. The center of the rectangular frame 261 coincides with the candidate 260.

[0131] Next, the non-maximum suppression processing unit 253 calculates the IoU of the rectangular frame 261 assigned to each candidate 260. As shown in Figure 35, for two rectangular frames 261 whose IoU is greater than or equal to a threshold (e.g., 0.3), the non-maximum suppression processing unit 253 leaves one representative rectangular frame 261 and assigns the other rectangular frame 261 to candidate 260. each Delete. This merges two rectangular frames 261 with IoU above the threshold into one rectangular frame 261. By deleting adjacent rectangular frames 261 and overlapping rectangular frames 261 and candidate 260 with IoU above the threshold in this way, a line image 258 representing the centerline of the aorta AO is finally obtained.

[0132] The conversion unit 254 generates line position information 259 based on the line image 258. The line position information 259 is the XYZ coordinates of multiple pixels representing the center line of the aorta AO in the line image 258.

[0133] In this way, the selection unit 252 selects elements from the probability distribution map 256 that have a probability of existence above a predetermined threshold as candidates 260 for the centerline of the aorta AO. The non-maximal suppression processing unit 253 assigns a rectangular frame 261 of a predetermined size to the selected candidates 260. Then, it applies non-maximal suppression processing to the rectangular frame 261 and identifies the centerline of the aorta AO based on the results of the non-maximal suppression processing. This method also allows for the accurate identification of lines such as the centerline of the aorta AO.

[0134] Note that the line is not limited to the centerline of the aorta (AO). It may also be the centerline of the ribs as seen in chest tomography image 190, or the centerline of the urethra as seen in lumbar tomography image, etc.

[0135] In each of the above embodiments, for example, the hardware structure of the Processing Unit that performs various processes such as the RW control unit 40, instruction receiving unit 41, extraction units 42, 165, 195, 210, 225, and 250, point position display map generation unit 43, object identification unit 44, anatomical naming unit 45, display control unit 46, analysis units 70, 166, 196, 211, 226, and 251, selection units 71, 212, and 252, non-maximum suppression processing units 72 and 253, conversion units 73 and 254, centroid calculation units 167 and 197, and thinning processing unit 227 can be any of the following types of processors. As mentioned above, the various types of processors include, in addition to the CPU 22, which is a general-purpose processor that executes software (operating program 30) and functions as various processing units, programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform particular processing.

[0136] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, and / or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0137] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, which then functions as multiple processing units, as exemplified by client and server computers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System-on-a-Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned processors.

[0138] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits, which are combinations of circuit elements such as semiconductor devices.

[0139] The images to be analyzed are not limited to tomographic images 15 obtained from a CT scanner 10. For example, tomographic images obtained from an MRI (Magnetic Resonance Imaging) scanner may also be used. Furthermore, the images to be analyzed are not limited to three-dimensional images such as tomographic images. For example, two-dimensional images such as simple radiographic images may also be used. Moreover, the images to be analyzed are not limited to medical images. Therefore, the objects to be analyzed are not limited to human body structures. For example, an image of a street scene may be used as the image to be analyzed, and a person's face may be used as the object.

[0140] The technology of this disclosure can be appropriately combined with the various embodiments and / or variations described above. Furthermore, it is understood that various configurations can be adopted without departing from the spirit of the invention, and the invention is not limited to the embodiments described above. Moreover, the technology of this disclosure extends not only to programs but also to storage media for storing programs non-temporarily.

[0141] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0142] In this specification, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0143] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

Claims

1. Processor and The processor comprises, The aforementioned processor, The semantic segmentation model is input to the image to be analyzed, and the semantic segmentation model outputs a feature map in the image to be analyzed that has features associated with one point corresponding to an object. The centroid of the feature map based on the aforementioned features is identified as one point within the image to be analyzed. Image processing device.

2. Processor and The processor comprises, The aforementioned processor, The semantic segmentation model is input to the image to be analyzed, and outputs a feature map which is a feature map having features related to discrete points corresponding to multiple objects in the image to be analyzed, and which is a probability distribution map whose features are the probability of existence of the discrete points. An output image is generated in which each pixel is labeled with a class corresponding to the existence probability of the probability distribution map. Based on the output image, identify the discrete points. Image processing device.

3. Processor and The processor comprises, The aforementioned processor, The semantic segmentation model is input to the image to be analyzed, and outputs a feature map which is a feature map having features related to discrete points corresponding to multiple objects in the image to be analyzed, and which is a probability distribution map whose features are the probability of existence of the discrete points. As candidates for the discrete points, elements with a predetermined probability of existence above a certain threshold are selected from each element of the probability distribution map. A pre-set rectangular frame is assigned to the selected candidate. The rectangular frame is subjected to non-maximum suppression processing, Based on the results of the non-maximal suppression process, identify the discrete points. Image processing device.

4. Processor and The processor comprises, The aforementioned processor, The semantic segmentation model is input to the image to be analyzed, and the semantic segmentation model outputs a feature map which is a feature map having features related to lines corresponding to objects having a linear structure in the image to be analyzed, and which is a probability distribution map whose features are the probability of existence of the lines. As candidates for the aforementioned line, elements with a predetermined probability of existence above a certain threshold are selected from each element of the probability distribution map. A pre-set rectangular frame is assigned to the selected candidate. The rectangular frame is subjected to non-maximum suppression processing, Based on the results of the non-maximum suppression process, the line is identified. Image processing device.

5. The processor is Using an annotation image in which a first pixel corresponding to one of the following—a single point corresponding to an object, a discrete set of points corresponding to multiple objects, or a line corresponding to an object having a linear structure—is assigned a first pixel value, and other second pixels are assigned second pixel values ​​different from the first pixel value, The first pixel is assigned a weight greater than that of the second pixel, and the loss is calculated. An image processing apparatus according to any one of claims 1 to 4, wherein the semantic segmentation model is learned based on the loss.

6. The images to be analyzed are medical images showing the inside of a patient's body. The image processing apparatus according to any one of claims 1 to 5, wherein the object is a structure of the body.

7. Inputting an image to be analyzed into a semantic segmentation model, and outputting a feature map from the semantic segmentation model having a feature associated with one point corresponding to an object in the image to be analyzed, and Identifying the centroid of the feature map based on the aforementioned features as one point within the image to be analyzed, A method for operating an image processing device, including the device itself.

8. Inputting an image to be analyzed into a semantic segmentation model, and outputting a feature map from the semantic segmentation model which is a feature map having features related to a plurality of discrete points corresponding to a plurality of objects in the image to be analyzed, and which is a probability distribution map having the probability of existence of the plurality of discrete points as the features, To generate an output image in which each pixel is labeled with a class corresponding to the existence probability of the aforementioned probability distribution map, and, Based on the output image, identify the discrete points. A method for operating an image processing device, including the device itself.

9. Inputting an image to be analyzed into a semantic segmentation model, and outputting a feature map from the semantic segmentation model which is a feature map having features related to a plurality of discrete points corresponding to a plurality of objects in the image to be analyzed, and which is a probability distribution map having the probability of existence of the plurality of discrete points as the features, As candidates for the discrete points, elements are selected from each element of the probability distribution map that have a probability of existence equal to or greater than a predetermined threshold. Assign a pre-set rectangular frame to the selected candidate. Applying non-maximum suppression processing to the aforementioned rectangular frame, and Based on the results of the non-maximal suppression process, identify the discrete points. A method for operating an image processing device, including the device itself.

10. Inputting an image to be analyzed into a semantic segmentation model, and outputting a feature map from the semantic segmentation model which is a feature map having features related to lines corresponding to objects having a linear structure in the image to be analyzed, and which is a probability distribution map having the probability of existence of the lines as the features, As candidates for the aforementioned line, elements having a pre-set probability of existence above a certain threshold are selected from each element of the probability distribution map. Assign a pre-set rectangular frame to the selected candidate. Applying non-maximum suppression processing to the aforementioned rectangular frame, and Based on the results of the non-maximum suppression process, identify the line. A method for operating an image processing device, including the device itself.

11. Inputting an image to be analyzed into a semantic segmentation model, and outputting a feature map from the semantic segmentation model having a feature associated with one point corresponding to an object in the image to be analyzed, Identifying the centroid of the feature map based on the aforementioned features as one point within the image to be analyzed, An operating program for an image processing device that causes a computer to perform a process including [specific operations].

12. Inputting an image to be analyzed into a semantic segmentation model, and outputting a feature map from the semantic segmentation model which is a feature map having features related to discrete points corresponding to multiple objects in the image to be analyzed, and which is a probability distribution map having the probability of existence of the discrete points as the features, To generate an output image in which each pixel is labeled with a class corresponding to the existence probability of the aforementioned probability distribution map, and, Based on the output image, identify the discrete points. An operating program for an image processing device that causes a computer to perform a process including [specific operations].

13. Inputting an image to be analyzed into a semantic segmentation model, and outputting a feature map from the semantic segmentation model which is a feature map having features related to a plurality of discrete points corresponding to a plurality of objects in the image to be analyzed, and which is a probability distribution map having the probability of existence of the plurality of discrete points as the features, As candidates for the discrete points, elements are selected from each element of the probability distribution map that have a probability of existence equal to or greater than a predetermined threshold. Assign a pre-set rectangular frame to the selected candidate. Applying non-maximum suppression processing to the aforementioned rectangular frame, and Based on the results of the non-maximal suppression process, identify the discrete points. An operating program for an image processing device that causes a computer to perform a process including [specific operations].

14. Inputting an image to be analyzed into a semantic segmentation model, and outputting a feature map from the semantic segmentation model which is a feature map having features related to lines corresponding to objects having a linear structure in the image to be analyzed, and which is a probability distribution map having the probability of existence of the lines as the features, As candidates for the aforementioned line, elements having a pre-set probability of existence above a certain threshold are selected from each element of the probability distribution map. Assign a pre-set rectangular frame to the selected candidate. Applying non-maximum suppression processing to the aforementioned rectangular frame, and Based on the results of the non-maximum suppression process, identify the line. An operating program for an image processing device that causes a computer to perform a process including [specific operations].

Images