Computer program, image processing device, image processing system, and image processing method

Abstract

To provide a computer program, a learning model generation method, an image processing device, an image processing system, and an image processing method that can accurately identify a muscle region.SOLUTION: A computer program causes a computer to perform processing of: acquiring a medical image including a muscle region; inputting, when the medical image is input, the acquired medical image to a second learning model that is trained to output region information of an entire muscle region; outputting the region information of the entire muscle region; generating an image of the entire muscle region based on the output region information of the entire muscle region; inputting, when inputting the medical image and the entire muscle region, the acquired medical image and the generated entire muscle region image to a first learning model that is trained to output region information for each muscle region of the muscle region; and outputting the region information for each muscle region.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a computer program, an image processing device, an image processing system, and an image processing method. [Background technology]

[0002] Compared to muscles in the forearm and lower leg, thigh muscles are larger, and increasing muscle mass can increase basal metabolic rate. The thigh is also an important part of the body for maintaining pelvic balance and supporting the knee joint, so understanding the condition of these muscles is essential for maintaining good health.

[0003] Patent Document 1 discloses an ultrasound diagnostic device that derives muscle weakness in a muscle to be measured by performing texture analysis on an ultrasound echo image obtained by irradiating the muscle to be measured with ultrasound. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-130596 Summary of the Invention [Problem to be solved by the invention]

[0005] A muscle is made up of multiple parts, each of which has an important role, and it is therefore desirable to be able to identify each part with high accuracy.

[0006] The present invention has been made in view of the above circumstances, and has as its object to provide a computer program, an image processing device, an image processing system, and an image processing method that can accurately identify muscle regions. [Means for solving the problem]

[0007] The present application includes multiple means for solving the above-mentioned problems, and as one example, a computer program causes a computer to execute the following process: acquire a medical image including a muscle part; input the acquired medical image into a second learning model that has been trained to output area information of the entire muscle part when a medical image is input, output area information of the entire muscle part; generate an entire muscle part image based on the output area information of the entire muscle part; and input the acquired medical image and the generated entire muscle part image into a first learning model that has been trained to output area information for each muscle part of the muscle part when a medical image and an entire muscle part image are input, output area information for each muscle part of the muscle part. [Effects of the Invention]

[0008] According to the present invention, muscle locations can be identified with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an image processing apparatus according to an embodiment of the present invention. [Figure 2] FIG. 2 illustrates an example of the configuration of a control unit. [Figure 3] FIG. 10 is a diagram illustrating an example of part detection by an image processing device. [Figure 4] FIG. 2 is a diagram illustrating an example of a method for generating a first learning model and a second learning model. [Figure 5] FIG. 10 is a diagram showing an example of femur contour detection by an image processing device. [Figure 6] FIG. 10 is a diagram illustrating an example of a method for generating an entire quadriceps muscle image. [Figure 7] FIG. 10 is a diagram showing a first example of a method for generating a fascia image. [Figure 8] FIG. 10 is a diagram showing a second example of a method for generating a fascia image. [Figure 9] FIG. 10 is a diagram showing an example of a region correction screen for each quadriceps muscle image (region information). [Figure 10] FIG. 10 is a diagram showing an example of a region correction screen for a fascia image (region information). [Figure 11]FIG. 10 is a diagram showing an example of region information for the entire quadriceps muscle output by a second learning model using a different algorithm. [Figure 12] FIG. 10 is a diagram illustrating another example of the configuration of the image processing device according to the present embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of a detection process for each quadriceps muscle performed by an image processing device. [Figure 14] FIG. 10 is a diagram illustrating an example of a learning model generation process performed by an image processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] An embodiment of the present invention will now be described. FIG. 1 is a diagram showing an example of the configuration of an image processing device 100 according to this embodiment. The image processing device 100 includes a control unit 10 that controls the entire device, a communication unit 11, a memory 12, a display unit 13, an operation unit 14, and a storage unit 19. The storage unit 19 stores a computer program 20, a first learning model 21, a second learning model 22, and a third learning model 23. The image processing device 100 may be configured as an image processing system by distributing each functional unit across multiple devices, or may be configured as one or more clouds (servers). For example, this system may include each unit implemented in a processor of a single computer, or may include multiple computers or clouds (servers) with each unit distributed across multiple devices. That is, this embodiment may be implemented using one or more processors.

[0011] 2 is a diagram showing an example of the configuration of the control unit 10. The control unit 10 includes an acquisition unit 31, an output unit 32, a reduction / enlargement processing unit 15, a contour extraction unit 16, an image generation unit 24, a relearning processing unit 17, a reliability index calculation unit 18, and a coincidence calculation unit 33. The reduction / enlargement processing unit 15 includes an interval determination unit 151. In this specification, the acquisition unit 31, the output unit 32, the reduction / enlargement processing unit 15, the contour extraction unit 16, the image generation unit 24, the relearning processing unit 17, the reliability index calculation unit 18, and the coincidence calculation unit 33 are also collectively referred to as the control unit 10.

[0012] The control unit 10 is configured to incorporate a required number of CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), etc. Furthermore, by executing a computer program 20, the control unit 10 can realize at least one function of the acquisition unit 31, output unit 32, reduction / enlargement processing unit 15, contour extraction unit 16, image generation unit 24, relearning processing unit 17, reliability index calculation unit 18, and matching calculation unit 33 by software instead of hardware. The control unit 10 performs processing using a first learning model 21, a second learning model 22, and a third learning model 23. Furthermore, the control unit 10 can execute processing defined by the computer program 20. In other words, the processing by the control unit 10 is also processing by the computer program 20.

[0013] The communication unit 11 includes, for example, a communication module and has a function of communicating with external devices via a communication network. The communication unit 11 can acquire medical images from devices such as an ultrasound diagnostic device, a computed tomography device, a magnetic resonance diagnostic device, an X-ray imaging device, an angiography X-ray diagnostic device, a PET examination device, or an electron microscope. The communication unit 11 stores the acquired medical images in the storage unit 19 and outputs them to the control unit 10. Note that in this specification, an ultrasound image will be described as an example of a medical image. Also, the quadriceps will be described as an example of a muscle part, each quadriceps will be described as an example of a muscle site, and the femur will be described as an example of a predetermined bone near the muscle part.

[0014] The memory 12 can be configured with semiconductor memory such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, etc. By expanding the computer program 20 in the memory 12, the control unit 10 can execute the computer program 20.

[0015] The display unit 13 can be configured with a liquid crystal panel, an organic EL (Electro Luminescence) display, or the like.

[0016] The operation unit 14 is configured with, for example, a hardware keyboard, a mouse, etc., and can be used to operate icons displayed on the display unit 13, move and operate a cursor, input characters, etc. The operation unit 14 may be configured with a touch panel.

[0017] The acquisition unit 31 of the control unit 10 acquires a thigh image including the thigh. The thigh image may be acquired from an external device (for example, an ultrasound diagnostic device) via the communication unit 11, or may be acquired from the storage unit 19. The thigh image is an image including the quadriceps (rectus femoris, vastus medialis, vastus intermedius, and vastus lateralis).

[0018] FIG. 3 is a diagram showing an example of region detection by the image processing device 100. In the example of FIG. 3, each quadriceps muscle (rectus femoris, vastus medialis, vastus intermedius, and vastus lateralis) is detected as a region. The control unit 10 inputs a thigh image to the second learning model 22. The second learning model 22 is trained to output region information of the entire quadriceps muscle when a thigh image is input. The output unit 32 of the control unit 10 can acquire region information of the entire quadriceps muscle from the second learning model 22 and output it to the image generation unit 24.

[0019] More specifically, when a thigh image is input, the second learning model 22 can output segmentation data (region information). The segmentation data is obtained by classifying each pixel of the thigh image into a class. The second learning model 22 can classify each pixel of the input thigh image into two classes, for example, class 0 and 1. Class 0 indicates the background and indicates the outer region of the quadriceps. Class 1 indicates the quadriceps.

[0020] The image generation unit 24 generates an entire quadriceps muscle image based on the region information of the entire quadriceps muscle output by the second learning model 22. The entire quadriceps muscle image is an image in which, for example, the region classified as class 0 is set to black and the region classified as class 1 is set to a specific color other than black.

[0021] Next, the control unit 10 inputs the thigh image and the generated entire quadriceps muscle image to the first learning model 21. The first learning model 21 has been trained to output region information for each quadriceps muscle (rectus femoris, vastus medialis, vastus intermedius, and vastus lateralis) when the thigh image and the entire quadriceps muscle image are input. The output unit 32 of the control unit 10 can acquire region information for each quadriceps muscle from the first learning model 21 and output it to the image generation unit 24.

[0022] More specifically, when a thigh image and an entire quadriceps muscle image are input, the first learning model 21 can output segmentation data (region information). The segmentation data is obtained by classifying each pixel of the thigh image into a class. The first learning model 21 can classify each pixel of the input thigh image into five classes, for example, classes 0, 2, 3, 4, and 5, within the entire quadriceps muscle region indicated by the entire quadriceps muscle image. Class 0 indicates the background and represents the outer region of the quadriceps muscle. Classes 2 to 5 indicate each part of the body. For example, class 2 indicates the rectus femoris muscle, class 3 indicates the vastus intermedius muscle, class 4 indicates the vastus lateralis muscle, and class 5 indicates the vastus medialis muscle.

[0023] Since the first learning model 21 receives not only thigh images but also entire quadriceps images, it can be trained to output area information for each quadriceps muscle without going outside the entire quadriceps muscle area shown in the entire quadriceps muscle image. This allows the muscle location (area of ​​each quadriceps muscle) to be identified with high accuracy.

[0024] The image generation unit 24 generates each quadriceps muscle image based on the region information of each quadriceps muscle output by the first learning model 21. For example, each quadriceps muscle image is an image in which regions classified into class 0 are set to black and regions (sites) classified into classes 2 to 5 are set to a different display mode (e.g., different colors) other than black (see FIGS. 7 and 8). Muscle mass may be calculated based on the size of each site, etc. By accurately detecting the region of each quadriceps muscle, the size of each quadriceps muscle can be accurately determined, and muscle mass can be accurately calculated.

[0025] Each of the first learning model 21 and the second learning model 22 may use, for example, semantic segmentation such as U-Net, Attention-Unet, or SegNet, or may use instance segmentation such as Mask R-CNN, DeepMask, or FCIS, or may be a GAN (Generative Adversarial Network).

[0026] FIG. 4 is a diagram showing an example of a method for generating the first learning model 21 and the second learning model 22. The second learning model 22 may be generated, for example, as follows: The control unit 10 acquires training data including thigh images and region information of the entire quadriceps. The training data may be collected and stored, for example, in an external server or the like, and acquired from the server. The control unit 10 inputs the thigh images included in the training data into the second learning model 22, and adjusts the parameters of the second learning model 22 so that the value of a loss function based on the region information of the entire quadriceps output by the second learning model 22 and the region information of the entire quadriceps as teacher data included in the training data is minimized.

[0027] The first learning model 21 may be generated, for example, as follows: The control unit 10 acquires training data including thigh images, entire quadriceps muscle images, and region information for each quadriceps muscle. The training data may be collected and stored, for example, in an external server or the like, and acquired from the server. The control unit 10 inputs the thigh images and entire quadriceps muscle images included in the training data into the first learning model 21, and adjusts the parameters of the first learning model 21 so that the value of a loss function based on the region information for each quadriceps muscle output by the first learning model 21 and the region information for each quadriceps muscle as teacher data included in the training data is minimized.

[0028] 5 is a diagram showing an example of femur contour detection by image processing device 100. Control unit 10 inputs a femur image to third learning model 23. Third learning model 23 is trained to output femur region information when a femur image is input. Output unit 32 of control unit 10 can acquire femur region information from third learning model 23 and output it to image generation unit 24.

[0029] More specifically, when a thigh image is input, the third learning model 23 can output segmentation data (region information). The segmentation data is obtained by classifying each pixel of the thigh image into a class. The third learning model 23 can classify each pixel of the input thigh image into two classes, for example, classes 10 and 11. Class 10 indicates the background and represents the outer region of the femur. Class 11 indicates the femur.

[0030] The image generation unit 24 generates a femur image based on the femur region information output by the third learning model 23. The femur image is an image in which, for example, regions classified into class 10 are set to black and regions classified into class 11 are set to a specific color other than black.

[0031] The contour extraction unit 16 extracts the contour of the femur based on the femur image generated by the image generation unit 24, and generates a contour image in which the contour of the femur is drawn. Contour extraction can be performed, for example, as follows: The femur image is converted into a binary image (0: background, 1: graphic pixel), the binary image is raster scanned, and the first detected graphic pixel is set as the starting point (first pixel). Eight neighboring pixels around the starting point pixel are examined, for example, counterclockwise, and if a graphic pixel is found, this is set as the second pixel. Eight neighboring pixels around the second pixel are examined, and if a graphic pixel is found, this is set as the third pixel. By repeating the same process thereafter, the pixels that make up the contour can be extracted. A library such as OpenCV may be used for contour extraction.

[0032] The third learning model 23 may be generated, for example, as follows: The control unit 10 acquires training data including femur images and femur region information. The femur region information may be, for example, labeled by labeling the entire femur region. While the contour of the femur may be labeled, labeling the contour and performing training may result in portions of the contour being cut off when detecting the femur contour. By labeling the entire femur region, the third learning model 23 detects a specific region, and extracting the contour from the detected region prevents portions of the contour from being cut off. The control unit 10 inputs femur images included in the training data to the third learning model 23 and adjusts the parameters of the third learning model 23 so that the value of a loss function based on the femur region information output by the third learning model 23 and the femur region information included in the training data as teacher data is minimized.

[0033] The third learning model 23 may be, for example, one that uses semantic segmentation such as U-Net, Attention-Unet, or SegNet, or one that uses instance segmentation such as Mask R-CNN, DeepMask, or FCIS, or may be a GAN (Generative Adversarial Network).

[0034] The first learning model 21, the second learning model 22, and the third learning model 23 may be generated by the image processing device 100, or an external learning processing device may generate each learning model, and the image processing device 100 may acquire each learning model from the learning processing device.

[0035] FIG. 6 is a diagram showing an example of a method for generating an entire quadriceps muscle image. In the case of FIG. 3, the image generation unit 24 is configured to generate an entire quadriceps muscle image based on region information of the entire quadriceps muscle. In the case of FIG. 6, the image generation unit 24 generates an entire quadriceps muscle image by further considering a contour image of the femur. In this case, the image generation unit 24 generates an entire quadriceps muscle image as follows. That is, the entire quadriceps muscle image is generated using, as a landmark, the contour line of the femur depicted in the contour image of the femur generated based on the region information of the femur output by the third learning model 23. For example, if a portion of the entire quadriceps muscle area is included inside the contour line of the femur, that portion is removed from the entire quadriceps muscle area. Furthermore, if the distance between the contour line and the entire quadriceps muscle area is greater than a predetermined threshold (number of pixels), the portion of the entire quadriceps muscle area facing the contour line may be expanded (enlarged) so that the distance between the contour line and the entire quadriceps muscle area is within the threshold. The image of the entire quadriceps serves as the basis for detecting the area information of each quadriceps (each quadriceps is detected individually without going beyond the range of the entire quadriceps), so it is preferable that the area of ​​the entire quadriceps is large (large area) within an appropriate range.

[0036] As described above, the control unit 10 inputs the acquired thigh image into the third learning model 23, which has been trained to output femur region information when a thigh image is input, outputs the femur region information, extracts the contour of the femur based on the output femur region information, and generates an entire quadriceps muscle image based on the output quadriceps muscle region information and the extracted contour.

[0037] FIG. 7 illustrates a first example of a method for generating a fascia image. A fascia image is an image obtained by removing each quadriceps muscle (muscle) from a thigh image or an entire quadriceps muscle image. The regions of each quadriceps muscle image are designated A, B, C, and D. In the case of the quadriceps muscle, regions A to D are the rectus femoris, vastus medialis, vastus intermedius, and vastus lateralis, respectively. The example in FIG. 7 illustrates a case in which the distance G between region A and region B is narrower than a predetermined range (a predetermined number of pixels). For example, a portion of region A and a portion of region B are in contact with each other. In this case, the fascia that would normally exist between region A and region B is not present. Therefore, the reduction / enlargement processing unit 15 performs a process of reducing at least one of region A and region B, thereby bringing the distance G between region A and region B within the predetermined range. The reduction process can be performed, for example, by reducing the pixels outside the region by approximately 1 to 2 pixels. The number of pixels to be reduced during the reduction process may be set to a desired number depending on the position of the region.

[0038] The image generating unit 24 can generate a fascia image by removing each quadriceps muscle image that has been subjected to reduction processing from the thigh image or the entire quadriceps muscle image, thereby enabling a user such as a doctor to generate an appropriate fascia image while viewing the image.

[0039] As described above, the interval determination unit 151 of the reduction / enlargement processing unit 15 determines whether the interval between each quadriceps muscle is within a predetermined range based on the region information of each quadriceps muscle. If the interval between the quadriceps muscles is narrower than the predetermined range, the reduction / enlargement processing unit 15 reduces at least one of the quadriceps muscles and corrects the region information. The output unit 32 can output fascia region information based on the region information of each quadriceps muscle, including the entire quadriceps muscle image and the corrected region information.

[0040] FIG. 8 is a diagram showing a second example of a method for generating a fascia image. The example in FIG. 8 shows a case where the distance G between region A and region D is wider than a predetermined range (a predetermined number of pixels). In this case, the area of ​​the fascia existing between region A and region D becomes large. Therefore, the reduction / enlargement processing unit 15 performs a process of enlarging (expanding) at least one of region A and region D, so that the distance G between region A and region D falls within the predetermined range. The enlargement process can be performed, for example, by increasing the pixels outside the region by about 1 to 2 pixels. Note that the number of pixels to be increased during the enlargement process may be set to a required number depending on the position of the region.

[0041] The image generating unit 24 can generate a fascia image by removing each quadriceps muscle image that has been enlarged from the thigh image or the entire quadriceps muscle image, thereby enabling a user such as a doctor to generate an appropriate fascia image while viewing the image.

[0042] As described above, the spacing determination unit 151 of the reduction / enlargement processing unit 15 determines whether the spacing between the quadriceps muscles is within a predetermined range based on the region information for each quadriceps muscle. If the spacing between the quadriceps muscles is wider than the predetermined range, the reduction / enlargement processing unit 15 corrects the region information by enlarging (expanding) at least one of the quadriceps muscles. The output unit 32 can output fascia region information based on the region information for each quadriceps muscle, including the entire quadriceps muscle image and the corrected region information.

[0043] 9 is a diagram showing an example of a region correction screen 210 for each quadriceps muscle image (region information). The region correction screen 210 displayed on the display unit 13 displays an image display field 214, an editing tool 211, an "Edit" icon 212, and a "Save" icon 213. The image display field 214 displays each quadriceps muscle image generated by the image generation unit 24 based on the region information of each quadriceps muscle output by the first learning model 21 (see FIG. 3). The regions of each quadriceps muscle are represented by A to D. Regions A to D are the rectus femoris, vastus medialis, vastus intermedius, and vastus lateralis, respectively.

[0044] A user such as a doctor operates "edit" icon 212, moves cursor 215 to the position of the desired part, and modifies the area of ​​the part using an editing tool in editing tools 211. The editing tools include icons for various tools, such as a brush for freely drawing lines by dragging on the image, a shape selection icon for drawing lines, curves, squares, circles, arrows, etc. on the image, a palette icon for changing the size, pattern, or color of the pen tip, and an eraser for erasing required parts of the image.

[0045] As shown in FIG. 9 , for example, the boundary 216 between part D and part C can be corrected to become boundary 217. When the correction is complete, the user can save the corrected quadriceps muscle images by operating the "save" icon 213. The storage destination may be the memory unit 19 or an external data server, etc. The corrected region information of each quadriceps muscle can be used as training data for re-learning the first learning model 21. The re-learning processing unit 17 can acquire the corrected region information of each quadriceps muscle and re-learn the first learning model 21.

[0046] As described above, the control unit 10 can accept corrections to the region information of each quadriceps muscle displayed on the display unit 13, and re-learn the first learning model 21 based on the thigh image and the entire quadriceps muscle image together with the region information for which corrections have been accepted. This can further improve the accuracy of detection of the region of each quadriceps muscle by the first learning model 21.

[0047] 10 is a diagram showing an example of a region correction screen 220 for a fascia image (region information). The region correction screen 220, displayed on the display unit 13, displays an image display field 221, editing tools 211, an "edit" icon 212, and a "save" icon 213. The fascia image generated by the image generation unit 24 is displayed in the image display field 221 (see FIGS. 7 and 8). A user such as a doctor operates the "edit" icon 212, moves the cursor 215 to the desired position, and uses the editing tools in the editing tools 211 to make corrections to the fascia image.

[0048] As shown in FIG. 10, for example, a portion 222 of the fascia can be modified as indicated by the reference numeral 223. This allows the fascia region to be modified. When the modification is complete, the user can save the modified fascia image (region information) by operating the "Save" icon 213. The storage destination may be the memory unit 19 or an external data server, etc. The modified fascia region information can be used as training data for re-learning the first learning model 21. The re-learning processing unit 17 can acquire the modified fascia region information and re-learn the first learning model 21 taking the fascia region information into consideration. In this case, the region information for each quadriceps muscle can be updated based on the modified fascia region information.

[0049] As described above, the control unit 10 can accept corrections to the fascia region information displayed on the display unit 13, and re-learn the first learning model 21 based on the thigh image and the entire quadriceps image together with the corrected region information (updated region information for each quadriceps muscle). This can further improve the detection accuracy of each quadriceps muscle region by the first learning model 21.

[0050] Next, a reliability index for the detection result of muscle parts by the image processing device 100 will be described.

[0051] FIG. 11 shows an example of quadriceps region information output by the second learning model 22 using different algorithms. The different algorithms are denoted by P1, P2, and P3. Examples of the algorithms include U-Net, Attention-Unet, SegNet, Mask R-CNN, DeepMask, FCIS, and GAN (Generative Adversarial Network). As shown in FIG. 11A, the second learning model 22 using algorithms P1 and P2 output quadriceps region information with a high degree of agreement (match rate). On the other hand, as shown in FIG. 11B, the second learning model 22 using algorithms P1 and P3 output quadriceps region information with a low degree of agreement (match rate). The reliability index calculation unit 18 can calculate the degree of agreement as the ratio of the number of pixels occupying the common quadriceps region to the number of pixels occupying the quadriceps region output by each of the second learning models 22 using algorithms P1 and P2. If both regions completely match, the degree of agreement is 100%.

[0052] The matching calculation unit 33 of the control unit 10 inputs the acquired thigh image into each of multiple types of second learning models 22 with different algorithms, and calculates the degree of matching of the region information of the entire quadriceps muscle output by each second learning model 22. If the calculated degree of matching is equal to or greater than a predetermined threshold, the image generation unit 24 may generate an entire quadriceps muscle image based on the output region information of the entire quadriceps muscle.

[0053] Furthermore, when the degree of agreement between the region information of the entire quadriceps muscle output by each of the multiple types of second learning models 22 is equal to or greater than a predetermined threshold (e.g., 85%, 90%, 95%, etc.), the region information of the entire quadriceps muscle with the highest degree of agreement may be used. Furthermore, among the degrees of agreement between the region information of the entire quadriceps muscle output by each of the multiple types of second learning models 22, the region information with the lowest degree of agreement may be excluded, and region information including all of the remaining region information may be used.

[0054] Alternatively, the confidence level (reliability of each pixel) of the region information of the entire quadriceps muscle output by the second learning model 22 may be averaged, and if the averaged confidence level is lower than a predetermined threshold, the second learning model 22 may be switched to one using a different algorithm to output the region information of the entire quadriceps muscle. Alternatively, the confidence level of each pixel may be weighted for each of multiple types of second learning models 22, and the model with the highest averaged confidence level may be used.

[0055] FIG. 12 is a diagram showing another example of the configuration of an image processing device 50 according to this embodiment. As shown in FIG. 12, the image processing device 50 may be, for example, a personal computer. The image processing device 50 may be configured with a CPU 51, a ROM 52, a RAM 53, an input unit 54, an output unit 55, and a recording medium reading unit 56. A computer program (computer program product) recorded on a recording medium 1 (e.g., an optically readable disk storage medium such as a CD-ROM) can be read by the recording medium reading unit 56 (e.g., an optical disk drive) and stored on a hard disk (not shown). Here, the computer program includes the processing procedures described in FIGS. 13 and 14 (described later). The computer program can be loaded into the RAM 53 and executed by the CPU 51.

[0056] 13 is a diagram showing an example of the detection process for each quadriceps muscle by the image processing device 100. For convenience, the following description will be given assuming that the control unit 10 is the main actor in the process. The control unit 10 acquires a thigh image (S11), inputs the thigh image to the second learning model 22, and acquires region information for the entire quadriceps muscle output by the second learning model 22 (see FIG. 3) (S12).

[0057] The control unit 10 inputs the thigh image into the third learning model 23, acquires the femur region information output by the third learning model 23 (S13), and generates a contour image of the femur based on the acquired femur region information (see Figure 5) (S14).

[0058] The control unit 10 generates an entire quadriceps muscle image based on the region information of the entire quadriceps muscle and the contour image of the femur (see FIG. 6) (S15). Note that the processes of steps S13 to S15 are not essential. In this case, the process of step S15 may be changed so that an entire quadriceps muscle image is generated based on the region information of the entire quadriceps muscle.

[0059] The control unit 10 inputs the thigh image and the entire quadriceps image to the first learning model 21, and acquires the region information of each quadriceps muscle output by the first learning model 21 (see FIG. 3) (S16).

[0060] The control unit 10 generates an image of each quadriceps muscle (S17) and determines whether the distance between each quadriceps muscle is within a predetermined range (S18). If the distance between each quadriceps muscle is within the predetermined range (YES in S18), a fascia image is generated (S19) and the process of step S21 described below is performed.

[0061] If the distance between the quadriceps is not within the predetermined range (NO in S18), the control unit 10 reduces or enlarges at least one of the adjacent quadriceps (see FIGS. 7 and 8) (S20) and performs the process of step S19. The control unit 10 displays the quadriceps muscle image and the fascia image on the display unit 13 (S21), and ends the process.

[0062] Fig. 14 is a diagram showing an example of a learning model generation process by the image processing device 100. Fig. 14 explains a method for generating the first learning model 21 and the second learning model 22. The control unit 10 acquires training data including a thigh image, region information of the entire quadriceps, and region information of each quadriceps (S31).

[0063] When a thigh image is input, the control unit 10 updates the internal parameters of the second learning model 22 so that the region information of the entire quadriceps muscle output by the second learning model 22 approaches the training data (see FIG. 4) (S32). The control unit 10 determines whether the value of the loss function is within an allowable range (S33), and if the value of the loss function is not within the allowable range (NO in S33), performs the processes from step S32 onwards.

[0064] If the value of the loss function is within the allowable range (YES in S33), the control unit 10 generates an entire quadriceps image based on the area information of the entire quadriceps (S34), and when a thigh image and an entire quadriceps image are input, the control unit 10 updates the internal parameters of the first learning model 21 so that the area information of each quadriceps output by the first learning model 21 approaches the training data (see Figure 4) (S35).

[0065] The control unit 10 determines whether the value of the loss function is within the allowable range (S36), and if the value of the loss function is not within the allowable range (NO in S36), performs the processes from step S35 onwards. If the value of the loss function is within the allowable range (YES in S36), the control unit 10 stores the generated first learning model 21 and second learning model 22 in the storage unit 19 (S37), and ends the process.

[0066] The computer program of this embodiment causes a computer to execute the following process: acquire a medical image including a muscle part; input the acquired medical image into a second learning model that has been trained to output area information of the entire muscle part when a medical image is input, output area information of the entire muscle part, generate an entire muscle part image based on the output area information of the entire muscle part; and input the acquired medical image and the generated entire muscle part image into a first learning model that has been trained to output area information for each muscle part of the muscle part when a medical image and an entire muscle part image are input, output area information for each muscle part of the muscle part.

[0067] The computer program of this embodiment causes a computer to execute the following process: when a medical image including a muscle part is input into a third learning model that has been trained to output area information of a specified bone near the muscle part, the acquired medical image is input into the third learning model, the model outputs area information of the specified bone, the contour of the bone is extracted based on the output area information of the specified bone, and an image of the entire muscle part is generated based on the area information of the entire muscle part output by the second learning model and the extracted contour.

[0068] The computer program of this embodiment causes a computer to execute the following process: input acquired medical images into each of multiple second learning models with different algorithms; calculate the degree of agreement of the area information of the entire muscle area output by each of the second learning models; and, if the calculated degree of agreement is equal to or greater than a predetermined threshold, generate an image of the entire muscle area based on the output area information of the entire muscle area.

[0069] The computer program of this embodiment causes the computer to execute a process of accepting corrections to the output region information for each muscle region, and re-learning the first learning model based on the medical image and the entire muscle image together with the region information for which corrections have been accepted.

[0070] The computer program of this embodiment causes a computer to execute the following process: determine whether the spacing between each muscle site is within a predetermined range based on the area information of each muscle site; if the spacing between the muscle sites is within the predetermined range, reduce the area information of at least one of the muscle sites and correct the area information; and output fascia area information based on the entire muscle image and the area information of each muscle site including the corrected area information.

[0071] The computer program of this embodiment causes the computer to execute a process in which, if the spacing between the muscle sites is wider than the specified range, the computer enlarges at least one of the muscle sites and corrects the area information, and outputs fascia area information based on the area information of each of the muscle sites including the entire muscle image and the corrected area information.

[0072] The computer program of this embodiment causes the computer to execute a process of accepting corrections to the output fascia area information and re-learning the first learning model based on the medical image and the entire muscle image together with the area information that has been corrected.

[0073] In the computer program of this embodiment, the muscle portion includes the thigh, the entire muscle portion includes the entire quadriceps, and the muscle site includes the quadriceps.

[0074] The learning model generation method of this embodiment acquires first training data including a medical image including a muscle part, an entire muscle part image showing the entire muscle part, and region information for each muscle part of the muscle part, and generates a first learning model based on the first training data so that when the medical image and the entire muscle part image are input, region information for each muscle part of the muscle part is output.

[0075] The learning model generation method of this embodiment acquires first training data including a medical image including a muscle part, area information of the entire muscle part, and area information for each muscle part of the muscle part; generates a second learning model based on the first training data so that when a medical image is input, area information of the entire muscle part is output; generates an image of the entire muscle part based on the area information of the entire muscle part; and generates a first learning model based on the first training data so that when the medical image and the generated image of the entire muscle part are input, area information for each muscle part of the muscle part is output.

[0076] The image processing device of this embodiment comprises an acquisition unit that acquires a medical image including a muscle part, a first output unit that inputs the acquired medical image into a second learning model that has been trained to output area information of the entire muscle part when a medical image is input, and outputs area information of the entire muscle part, an image generation unit that generates an image of the entire muscle part based on the output area information of the entire muscle part, and a second output unit that inputs the acquired medical image and the generated image of the entire muscle part into the first learning model that has been trained to output area information of each muscle part of the muscle part when a medical image and an image of the entire muscle part are input, and outputs area information of each muscle part.

[0077] The image processing system of this embodiment comprises an acquisition unit that acquires a medical image including a muscle part, a first output unit that inputs the acquired medical image into a second learning model that has been trained to output area information of the entire muscle part when a medical image is input, and outputs area information of the entire muscle part, an image generation unit that generates an image of the entire muscle part based on the output area information of the entire muscle part, and a second output unit that inputs the acquired medical image and the generated image of the entire muscle part into the first learning model that has been trained to output area information of each muscle part of the muscle part when a medical image and an image of the entire muscle part are input, and outputs area information of each muscle part.

[0078] In the image processing method of this embodiment, a medical image including a muscle part is acquired, the acquired medical image is input into a second learning model that has been trained to output area information of the entire muscle part when a medical image is input, and area information of the entire muscle part is output, and an image of the entire muscle part is generated based on the output area information of the entire muscle part, and the acquired medical image and the generated image of the entire muscle part are input into a first learning model that has been trained to output area information for each muscle part of the muscle part when a medical image and an image of the entire muscle part are input, and area information for each muscle part is output.

[0079] In this embodiment, the quadriceps of the thigh are described as the muscle part, and each quadriceps is described as the muscle site, but the muscle part is not limited to the muscles of the thigh. For example, the muscle part may be the upper arm, abdomen, forearm, or lower leg, the muscle site may be each muscle that constitutes each muscle part, and the predetermined bone near the muscle part may be a bone near each muscle part. [Explanation of symbols]

[0080] 1. Recording media 10 Control Unit 11 Communications Department 12 Memory 13 Display section 14 Control section 15 Reduction / enlargement processing section 151 Interval judgment section 16 Contour extraction section 17 Re-learning processing unit 18 Reliability index calculation unit 19 Memory section 20 Computer Programs 21 First Learning Model 22 Second Learning Model 23 Third Learning Model 24 Image generation unit 31 Acquisition Department 32 Output section 33 Matching degree calculation part 51 CPU 52 ROM 53 RAM 54 Input section 55 Output section 56 Recording medium reading unit 100, 50 Image processing device

Claims

1. On the computer, Acquire a medical image including a muscle area, When a medical image is input, the acquired medical image is input to a second learning model that has been trained to output region information of the entire muscle part, and region information of the entire muscle part is output; An image of the entire muscle is generated based on the outputted region information of the entire muscle. a first learning model that has been trained to output region information for each muscle part when a medical image and an entire muscle part image are input, the acquired medical image and the generated entire muscle part image are input, and the first learning model outputs region information for each muscle part; determining whether the interval between each muscle site is within a predetermined range based on the area information of each muscle site; If the distance between the muscle sites is within the predetermined range, at least one of the muscle sites is reduced to correct the region information; outputting fascia region information based on the muscle region overall image and the region information of each muscle region including the corrected region information; A computer program that executes a process.

2. On the computer, Acquire a medical image including a muscle area, When a medical image is input, the acquired medical image is input to a second learning model that has been trained to output region information of the entire muscle part, and region information of the entire muscle part is output; An image of the entire muscle is generated based on the outputted region information of the entire muscle. a first learning model that has been trained to output region information for each muscle part when a medical image and an entire muscle part image are input, the acquired medical image and the generated entire muscle part image are input, and the first learning model outputs region information for each muscle part; determining whether the interval between each muscle site is within a predetermined range based on the area information of each muscle site; If the distance between the muscle sites is greater than the predetermined range, at least one of the muscle sites is enlarged to correct the region information; outputting fascia region information based on the muscle region overall image and the region information of each muscle region including the corrected region information; A computer program that executes a process.

3. On the computer, inputting the acquired medical image into each of the plurality of second learning models having different algorithms, and calculating the degree of agreement of the region information of the entire muscle portion output by each of the second learning models; If the calculated degree of coincidence is equal to or greater than a predetermined threshold, an image of the entire muscle part is generated based on the output region information of the entire muscle part.

3. The computer program according to claim 1, which executes a process.

4. On the computer, Accepting corrections to the outputted area information for each of the muscle regions; re-learning the first learning model based on the medical image and the entire muscle image together with the corrected region information; The computer program according to any one of claims 1 to 3, which causes a process to be executed.

5. On the computer, Accepts corrections to the output fascia area information, re-learning the first learning model based on the medical image and the entire muscle image together with the corrected region information; 5. A computer program product according to claim 1, which causes a process to be executed.

6. the muscle portion includes a thigh portion; the entire muscle section includes the entire quadriceps; The muscle site includes the quadriceps. A computer program according to any one of claims 1 to 5.

7. an acquisition unit for acquiring medical images including muscle areas; a first output unit that inputs the acquired medical image into a second learning model that has been trained to output area information of the entire muscle region when the medical image is input, and outputs area information of the entire muscle region; an image generation unit that generates an image of the entire muscle based on the output area information of the entire muscle; a second output unit that inputs the acquired medical image and the generated overall image of the muscle portion into a first learning model that has been trained to output area information for each muscle part when a medical image and an overall image of the muscle part are input, and outputs area information for each muscle part; a determination unit that determines whether the intervals between the muscle sites are within a predetermined range based on the area information of each muscle site; a correction unit that corrects region information by reducing at least one of the muscle regions when the distance between the muscle regions is within the predetermined range; a third output unit that outputs fascia region information based on the muscle region overall image and the region information of each muscle region including the corrected region information; Equipped with Image processing device.

8. an acquisition unit for acquiring medical images including muscle areas; a first output unit that inputs the acquired medical image into a second learning model that has been trained to output area information of the entire muscle region when the medical image is input, and outputs area information of the entire muscle region; an image generation unit that generates an image of the entire muscle based on the output area information of the entire muscle; a second output unit that inputs the acquired medical image and the generated overall image of the muscle portion into a first learning model that has been trained to output area information for each muscle part when a medical image and an overall image of the muscle part are input, and outputs area information for each muscle part; a determination unit that determines whether the intervals between the muscle sites are within a predetermined range based on the area information of each muscle site; a correction unit that corrects area information by enlarging at least one of the muscle parts when the distance between the muscle parts is wider than the predetermined range; a third output unit that outputs fascia region information based on the muscle region overall image and the region information of each muscle region including the corrected region information; Equipped with Image processing device.

9. an acquisition unit for acquiring medical images including muscle areas; a first output unit that inputs the acquired medical image into a second learning model that has been trained to output area information of the entire muscle region when the medical image is input, and outputs area information of the entire muscle region; an image generation unit that generates an image of the entire muscle based on the output area information of the entire muscle; a second output unit that inputs the acquired medical image and the generated overall image of the muscle portion into a first learning model that has been trained to output area information for each muscle part when a medical image and an overall image of the muscle part are input, and outputs area information for each muscle part; a determination unit that determines whether the intervals between the muscle sites are within a predetermined range based on the area information of each muscle site; a correction unit that corrects region information by reducing at least one of the muscle regions when the distance between the muscle regions is within the predetermined range; a third output unit that outputs fascia region information based on the muscle region overall image and the region information of each muscle region including the corrected region information; Equipped with Image processing system.

10. an acquisition unit for acquiring medical images including muscle areas; a first output unit that inputs the acquired medical image into a second learning model that has been trained to output area information of the entire muscle region when the medical image is input, and outputs area information of the entire muscle region; an image generation unit that generates an image of the entire muscle based on the output area information of the entire muscle; a second output unit that inputs the acquired medical image and the generated overall image of the muscle portion into a first learning model that has been trained to output area information for each muscle part when a medical image and an overall image of the muscle part are input, and outputs area information for each muscle part; a determination unit that determines whether the intervals between the muscle sites are within a predetermined range based on the area information of each muscle site; a correction unit that corrects area information by enlarging at least one of the muscle parts when the distance between the muscle parts is wider than the predetermined range; a third output unit that outputs fascia region information based on the muscle region overall image and the region information of each muscle region including the corrected region information; Equipped with Image processing system.

11. Acquire a medical image including a muscle area, When a medical image is input, the acquired medical image is input to a second learning model that has been trained to output region information of the entire muscle part, and region information of the entire muscle part is output; An image of the entire muscle is generated based on the outputted region information of the entire muscle. a first learning model that has been trained to output region information for each muscle part when a medical image and an entire muscle part image are input, the acquired medical image and the generated entire muscle part image are input, and the first learning model outputs region information for each muscle part; determining whether the interval between each muscle site is within a predetermined range based on the area information of each muscle site; If the distance between the muscle sites is within the predetermined range, at least one of the muscle sites is reduced to correct the region information; outputting fascia region information based on the muscle region overall image and the region information of each muscle region including the corrected region information; Image processing methods.

12. Acquire a medical image including a muscle area, When a medical image is input, the acquired medical image is input to a second learning model that has been trained to output region information of the entire muscle part, and region information of the entire muscle part is output; An image of the entire muscle is generated based on the outputted region information of the entire muscle. a first learning model that has been trained to output region information for each muscle part when a medical image and an entire muscle part image are input, the acquired medical image and the generated entire muscle part image are input, and the first learning model outputs region information for each muscle part; determining whether the interval between each muscle site is within a predetermined range based on the area information of each muscle site; If the distance between the muscle sites is greater than the predetermined range, at least one of the muscle sites is enlarged to correct the region information; outputting fascia region information based on the muscle region overall image and the region information of each muscle region including the corrected region information; Image processing methods.

Images