Image processing system, image processing method, image processing program, and image processing device

Abstract

The present invention addresses the problem of deciding an interest region more accurately. An image processing system 1 comprises: a landmark detection unit 43 that detects two or more joint landmarks in a joint image, which is an internal image of a joint of a human, the joint landmarks being landmarks related to the joint; and an ROI 1 decision unit 44 that decides the interest region related to the joint in the joint image on the basis of the detected two or more joint landmarks. The landmark detection unit 43 may be detected by using a landmark detection model that detects the two or more joint landmarks in the joint image upon the joint image being input thereto. The landmark detection model may be a model trained on the basis of learning data composed of sets of the joint images and images obtained by annotating the two or more joint landmarks in the joint images.

Description

Image processing system, image processing method, image processing program, and image processing device

[0001] One aspect of the present disclosure relates to an image processing system, an image processing method, an image processing program, and an image processing device that process images.

[0002] Japanese Patent Application Laid-Open No. 2003-144992 discloses an information processing device that sets, for each joint of a person, a square of a predetermined size centered on the joint as a region of interest.

[0003] Japanese Patent Application Laid-Open No. 2022-080113

[0004] In the above-described information processing device, a square of a predetermined size centered on a joint is set as a region of interest, which may result in the region of interest not accurately capturing the joint. Therefore, it is desirable to determine a more accurate region of interest.

[0005] An image processing system according to one aspect of the present disclosure includes a landmark detection unit that detects two or more joint landmarks that are landmarks related to a joint in a joint image that is an in-vivo image of a human joint, and a determination unit that determines a region of interest related to the joint in the joint image based on the two or more detected joint landmarks. In this aspect, the region of interest is determined based on the two or more detected joint landmarks. This allows for more accurate determination of the region of interest.

[0006] The landmark detection unit may detect the joint landmarks using a landmark detection model that detects two or more joint landmarks in the joint image when the joint image is input. In this aspect, by using the landmark detection model, it is possible to more reliably and accurately detect two or more joint landmarks.

[0007] The landmark detection model may be a model trained based on training data consisting of pairs of the joint image and an image in which two or more of the joint landmarks in the joint image are annotated. In this aspect, the landmark detection model trained based on the training data can detect two or more joint landmarks more reliably and accurately.

[0008] The landmark detection unit may also detect the type of each of the joint landmarks, and the determination unit may make the determination based further on the type of each of the detected joint landmarks. In this aspect, the region of interest is determined based further on the type of each of the joint landmarks. This allows for a more accurate determination of the region of interest.

[0009] One of the two or more detected joint landmarks may be a bone, and the remaining may be a bone other than the bone, cartilage, fat, joint capsule, or muscle. In such an aspect, a more accurate region of interest can be determined based on a more specific joint landmark.

[0010] The joint may be a joint of a hemophilia patient. In this aspect, a region of interest in an image of a joint of a hemophilia patient may be determined.

[0011] The joint image may be an ultrasound image or an X-ray image. Generally, ultrasound examination or X-ray examination can be easily performed in an outpatient setting, so in this aspect, for example, the region of interest can be easily determined in an outpatient setting.

[0012] The joint image may be an image with enhanced image quality. In this aspect, two or more joint landmarks can be detected more accurately based on the image with enhanced image quality, thereby enabling more accurate determination of the region of interest.

[0013] The joint landmark may be a tibia, talus, fibula, femur, patella, humerus, radius, ulna, cartilage, fat, joint capsule, or muscle. In this aspect, a more accurate region of interest can be determined based on a more specific joint landmark.

[0014] The determining unit may determine the region of interest based on a distance from the center of the joint landmark to a boundary line of the region of interest. In this aspect, the region of interest can be determined more reliably.

[0015] The apparatus may further include a state detection unit that detects a joint state, which is a state of the joint, based on a local joint image that is an image within the determined region of interest among the joint images. In this aspect, the joint state can be detected more accurately based on the local joint image, thereby allowing the user to more accurately understand the joint state.

[0016] The state detection unit may detect the joint state of the joint indicated by the local joint image using a state detection model when the local joint image is input. In this aspect, the use of the state detection model makes it possible to more reliably and accurately detect the joint state.

[0017] The condition detection model may be a model trained based on training data consisting of pairs of the local joint image and a local joint image associated with a normal label or a local joint image associated with an abnormal location of the joint indicated by the local joint image, In this aspect, the condition detection model trained based on the training data can detect the joint condition more reliably and accurately.

[0018] The present invention may further include a visualization unit that visualizes, in a display mode according to a degree of influence, a portion of the local joint image that has influenced the state detection model when detecting the joint state. In this aspect, the user can grasp the joint state in more detail from the display mode.

[0019] The joint condition may be indicated as normal, abnormal, bleeding, synovitis, or arthropathy. In this aspect, a more specific joint condition can be output, allowing the user to understand the more specific joint condition.

[0020] When the joint condition indicates abnormality, bleeding, synovitis, or arthropathy, the corresponding location in the local joint image may be further displayed. In this aspect, the corresponding location in the local joint image may be further displayed in addition to displaying the abnormality, bleeding, synovitis, or arthropathy. This allows the user to understand the joint condition more specifically.

[0021] The joint may be an ankle joint, the joint landmark may be a tibia, a fibula, a talus, or fat, and the joint status may indicate normal, abnormal, bleeding, synovitis, or arthropathy. In this aspect, a more accurate region of interest can be determined for the ankle joint image based on two or more specific joint landmarks. A specific joint status can then be output based on the determined region of interest. This allows the user to grasp the more accurate and specific joint status of the ankle joint.

[0022] The joint may be a knee joint, the joint landmark may be a femur, a tibia, a fibula, a patella, or fat, and the joint status may indicate normal, abnormal, bleeding, synovitis, or arthropathy. In this aspect, a more accurate region of interest can be determined for the knee joint image based on two or more specific joint landmarks. Then, a specific joint status can be output based on the determined region of interest. This allows the user to grasp the more accurate and specific joint status of the knee joint.

[0023] The joint may be an elbow joint, the joint landmark may be a humerus, a radius, an ulna, or fat, and the joint condition may indicate normal, abnormal, bleeding, synovitis, or arthropathy. In this aspect, a more accurate region of interest may be determined for the elbow joint image based on two or more specific joint landmarks. A specific joint condition may then be output based on the determined region of interest. This allows the user to grasp the more accurate and specific joint condition of the elbow joint.

[0024] A diagnosis of the joint condition may be assisted based on the joint image. In this aspect, a diagnosis of the joint condition can be assisted based on the joint image.

[0025] An image processing program according to one aspect of the present disclosure causes a computer to function as a landmark detection unit that detects two or more joint landmarks related to a joint in a joint image that is an in-vivo image of a human joint, and a determination unit that determines a region of interest related to the joint in the joint image based on the two or more detected joint landmarks. In this aspect, the region of interest is determined based on the two or more detected joint landmarks. This allows for more accurate determination of the region of interest.

[0026] The landmark detection unit may detect the joint landmarks using a landmark detection model that detects two or more joint landmarks in the joint image when the joint image is input. In this aspect, by using the landmark detection model, it is possible to more reliably and accurately detect two or more joint landmarks.

[0027] The landmark detection model may be a model trained based on training data consisting of pairs of the joint image and an image in which two or more of the joint landmarks in the joint image are annotated. In this aspect, the landmark detection model trained based on the training data can detect two or more joint landmarks more reliably and accurately.

[0028] The landmark detection unit may also detect the type of each of the joint landmarks, and the determination unit may make the determination based further on the type of each of the detected joint landmarks. In this aspect, the region of interest is determined based further on the type of each of the joint landmarks. This allows for a more accurate determination of the region of interest.

[0029] One of the two or more detected joint landmarks may be a bone, and the remaining may be a bone other than the bone, cartilage, fat, joint capsule, or muscle. In such an aspect, a more accurate region of interest can be determined based on a more specific joint landmark.

[0030] The joint may be a joint of a hemophilia patient. In this aspect, a region of interest in an image of a joint of a hemophilia patient may be determined.

[0031] The joint image may be an ultrasound image or an X-ray image. Generally, ultrasound examination or X-ray examination can be easily performed in an outpatient setting, so in this aspect, for example, the region of interest can be easily determined in an outpatient setting.

[0032] The joint image may be an image with enhanced image quality. In this aspect, two or more joint landmarks can be detected more accurately based on the image with enhanced image quality, thereby enabling more accurate determination of the region of interest.

[0033] The joint landmark may be a tibia, talus, fibula, femur, patella, humerus, radius, ulna, cartilage, fat, joint capsule, or muscle. In this aspect, a more accurate region of interest can be determined based on a more specific joint landmark.

[0034] The determining unit may determine the region of interest based on a distance from the center of the joint landmark to a boundary line of the region of interest. In this aspect, the region of interest can be determined more reliably.

[0035] The computer may further function as a condition detection unit that detects a joint condition, which is a condition of the joint, based on a local joint image, which is an image within the determined region of interest of the joint image. In this aspect, the joint condition can be detected more accurately based on the local joint image, thereby allowing the user to more accurately understand the joint condition.

[0036] The state detection unit may detect the joint state of the joint indicated by the local joint image using a state detection model when the local joint image is input. In this aspect, the use of the state detection model makes it possible to more reliably and accurately detect the joint state.

[0037] The condition detection model may be a model trained based on training data consisting of pairs of the local joint image and a local joint image associated with a normal label of the joint indicated by the local joint image or a local joint image associated with an abnormal location of the joint indicated by the local joint image. In this aspect, the condition detection model trained based on the training data can detect the joint condition more reliably and accurately.

[0038] The computer may further function as a visualization unit that visualizes, in a display mode according to the degree of influence, a portion of the local joint image that has influenced the state detection model when the state detection model detects the joint state. In this aspect, the user can grasp the joint state in more detail from the display mode.

[0039] The joint condition may be indicated as normal, abnormal, bleeding, synovitis, or arthropathy. In this aspect, a more specific joint condition can be output, allowing the user to understand the more specific joint condition.

[0040] When the joint condition indicates abnormality, bleeding, synovitis, or arthropathy, the corresponding location in the local joint image may be further displayed. In this aspect, the corresponding location in the local joint image may be further displayed in addition to displaying the abnormality, bleeding, synovitis, or arthropathy. This allows the user to understand the joint condition more specifically.

[0041] The joint may be an ankle joint, the joint landmark may be a tibia, a fibula, a talus, or fat, and the joint status may indicate normal, abnormal, bleeding, synovitis, or arthropathy. In this aspect, a more accurate region of interest can be determined for the ankle joint image based on two or more specific joint landmarks. A specific joint status can then be output based on the determined region of interest. This allows the user to grasp the more accurate and specific joint status of the ankle joint.

[0042] The joint may be a knee joint, the joint landmark may be a femur, a tibia, a fibula, a patella, or fat, and the joint status may indicate normal, abnormal, bleeding, synovitis, or arthropathy. In this aspect, a more accurate region of interest can be determined for the knee joint image based on two or more specific joint landmarks. Then, a specific joint status can be output based on the determined region of interest. This allows the user to grasp the more accurate and specific joint status of the knee joint.

[0043] The joint may be an elbow joint, the joint landmark may be a humerus, a radius, an ulna, or fat, and the joint condition may indicate normal, abnormal, bleeding, synovitis, or arthropathy. In this aspect, a more accurate region of interest may be determined for the elbow joint image based on two or more specific joint landmarks. A specific joint condition may then be output based on the determined region of interest. This allows the user to grasp the more accurate and specific joint condition of the elbow joint.

[0044] A diagnosis of the joint condition may be assisted based on the joint image. In this aspect, a diagnosis of the joint condition can be assisted based on the joint image.

[0045] An image processing method according to one aspect of the present disclosure is an image processing method executed by a computer, and includes a landmark detection step of detecting two or more joint landmarks related to a joint in a joint image, which is an in-vivo image of a human joint, and a determination step of determining a region of interest related to the joint in the joint image based on the two or more detected joint landmarks. In this aspect, the region of interest is determined based on the two or more detected joint landmarks. This allows for more accurate determination of the region of interest.

[0046] An image processing device according to an aspect of the present disclosure includes a landmark detection unit that detects two or more joint landmarks that are landmarks related to a joint in a joint image that is an in-vivo image of a human joint, and a determination unit that determines a region of interest related to the joint in the joint image based on the two or more detected joint landmarks. In this aspect, the region of interest is determined based on the two or more detected joint landmarks. This allows for more accurate determination of the region of interest.

[0047] The apparatus may further include a state detection unit that detects a joint state, which is a state of the joint, based on a local joint image that is an image within the determined region of interest among the joint images. In this aspect, the joint state can be detected more accurately based on the local joint image, thereby allowing the user to more accurately understand the joint state.

[0048] An image processing system, an image processing method, an image processing program, and an image processing device according to an aspect of the present disclosure can also be described as follows. [1] An image processing system comprising: a landmark detection unit that detects two or more joint landmarks that are landmarks related to a joint in a joint image that is an in-vivo image of a human joint; and a determination unit that determines a region of interest related to the joint in the joint image based on the two or more detected joint landmarks. [2] The image processing system according to [1], wherein the landmark detection unit detects the joint landmarks using a landmark detection model that detects two or more joint landmarks in the joint image when the joint image is input. [3] The image processing system according to [2], wherein the landmark detection model is a model trained based on training data consisting of pairs of the joint image and images in which the two or more joint landmarks in the joint image are annotated. [4] The image processing system according to any one of [1] to [3], wherein the landmark detection unit detects the type of each joint landmark, and the determination unit makes a determination further based on the type of each detected joint landmark. [5] The image processing system according to any one of [1] to [4], wherein one of the two or more detected joint landmarks is a bone, and the remaining are a bone other than the bone, cartilage, fat, joint capsule, or muscle. [6] The image processing system according to any one of [1] to [5], wherein the joint is a joint of a hemophilia patient. [7] The image processing system according to any one of [1] to [6], wherein the joint image is an ultrasound image or an X-ray image. [8] The image processing system according to any one of [1] to [7], wherein the joint image is an image with enhanced image quality. [9] The image processing system according to any one of [1] to [8], wherein the joint landmarks are a tibia, talus, fibula, femur, patella, humerus, radius, ulna, cartilage, fat, joint capsule, or muscle.

[10] The image processing system according to any one of [1] to [9], wherein the determination unit determines the region of interest based on a distance from a center of the joint landmark to a boundary line of the region of interest.

[11] The image processing system according to any one of [1] to

[10] , further comprising a state detection unit that detects a joint state, which is a state of the joint, based on a local joint image that is an image within the determined region of interest among the joint images.

[12] The image processing system according to

[11] , wherein the state detection unit, when inputting the local joint image, detects the joint state of the joint indicated by the local joint image using a state detection model that detects the joint state of the joint indicated by the local joint image.

[13] The image processing system according to

[12] , wherein the state detection model is a model trained based on training data consisting of pairs of the local joint image and the local joint image associated with a normal label or the local joint image associated with an abnormal location of the joint indicated by the local joint image.

[14] The image processing system according to

[12] or

[13] , further comprising a visualization unit that visualizes a portion of the local joint image that affected the detection of the joint state by the state detection model in a display mode according to the degree of influence.

[15] The image processing system according to any one of

[11] to

[14] , wherein the joint status indicates normal, abnormal, bleeding, synovitis, or arthropathy.

[16] The image processing system according to

[15] , wherein, when the joint status indicates abnormal, bleeding, synovitis, or arthropathy, the image processing system further indicates a corresponding location in the local joint image.

[17] The image processing system according to any one of

[11] to

[16] , wherein the joint is an ankle joint, the joint landmark is a tibia, fibula, talus, or fat, and the joint status indicates normal, abnormal, bleeding, synovitis, or arthropathy.

[18] The image processing system according to any one of

[11] to

[16] , wherein the joint is a knee joint, the joint landmark is a femur, tibia, fibula, patella, or fat, and the joint status indicates normal, abnormal, bleeding, synovitis, or arthropathy.

[19] The image processing system according to any one of

[11] to

[16] , wherein the joint is an elbow joint, the joint landmark is a humerus, a radius, an ulna, or fat, and the joint state indicates normal, abnormal, bleeding, synovitis, or arthropathy.

[20] The image processing system according to any one of

[11] to

[19] , which assists in diagnosing the joint condition based on the joint image.

[21] An image processing program for causing a computer to function as: a landmark detection unit that detects two or more joint landmarks that are landmarks related to the joint in a joint image that is an in-vivo image of a human joint; and a determination unit that determines a region of interest related to the joint in the joint image based on the two or more detected joint landmarks.

[22] The image processing program according to

[21] , wherein the landmark detection unit detects the joint landmarks using a landmark detection model that detects two or more joint landmarks in the joint image when the joint image is input.

[23] The image processing program according to

[22] , wherein the landmark detection model is a model trained based on training data consisting of pairs of the joint image and images in which the two or more joint landmarks in the joint image are annotated.

[24] The image processing program according to any one of

[21] to

[23] , wherein the landmark detection unit detects the type of each of the joint landmarks, and the determination unit makes a determination further based on the type of each of the detected joint landmarks.

[25] The image processing program according to any one of

[21] to

[24] , wherein one of the two or more detected joint landmarks is a bone, and the remaining are a bone other than the bone, cartilage, fat, joint capsule, or muscle.

[26] The image processing program according to any one of

[21] to

[25] , wherein the joint is a joint of a hemophilia patient.

[27] The image processing program according to any one of

[21] to

[26] , wherein the joint image is an ultrasound image or an X-ray image.

[28] The image processing program according to any one of

[21] to

[27] , wherein the joint image is an image with enhanced image quality.

[29] The image processing program according to any one of

[21] to

[28] , wherein the joint landmark is a tibia, a talus, a fibula, a femur, a patella, a humerus, a radius, an ulna, cartilage, fat, a joint capsule, or a muscle.

[30] The image processing program according to any one of

[21] to

[29] , wherein the determination unit determines the region of interest based on a distance from the center of the joint landmark to a boundary line of the region of interest.

[31] The image processing program according to any one of

[21] to

[30] , for causing the computer to further function as a condition detection unit that detects a joint state, which is a state of the joint, based on a local joint image that is an image within the determined region of interest among the joint images.

[32] The image processing program according to

[31] , wherein the condition detection unit, when inputting the local joint image, detects the joint state of the joint indicated by the local joint image using a condition detection model that detects the joint state of the joint indicated by the local joint image.

[33] The image processing program according to

[32] , wherein the condition detection model is a model trained based on training data consisting of pairs of the local joint image and the local joint image associated with a normal label or the local joint image associated with an abnormal location of the joint indicated by the local joint image.

[34] The image processing program according to

[32] or

[33] , for causing the computer to further function as a visualization unit that visualizes, in a display mode according to the degree of influence, a portion of the local joint image that has affected the condition detection model when detecting the joint condition.

[35] The image processing program according to any one of

[31] to

[34] , which indicates that the joint condition is normal, abnormal, bleeding, synovitis, or arthropathy.

[36] The image processing program according to

[35] , which, when the joint condition indicates abnormal, bleeding, synovitis, or arthropathy, further indicates the corresponding location in the local joint image.

[37] The image processing program according to any one of

[31] to

[36] , wherein the joint is an ankle joint, the joint landmark is a tibia, fibula, talus, or fat, and the joint condition indicates that it is normal, abnormal, bleeding, synovitis, or arthropathy.

[38] The image processing program according to any one of

[31] to

[36] , wherein the joint is a knee joint, the joint landmark is a femur, a tibia, a fibula, a patella, or fat, and the joint state indicates normal, abnormal, bleeding, synovitis, or arthropathy.

[39] The image processing program according to any one of

[31] to

[36] , wherein the joint is an elbow joint, the joint landmark is a humerus, a radius, an ulna, or fat, and the joint state indicates normal, abnormal, bleeding, synovitis, or arthropathy.

[40] The image processing program according to any one of

[31] to

[39] , which supports diagnosis of the joint state based on the joint image.

[41] An image processing method executed by a computer, comprising: a landmark detection step of detecting two or more joint landmarks that are landmarks related to the joint in a joint image that is an in-vivo image of a human joint; and a determination step of determining a region of interest related to the joint in the joint image based on the two or more detected joint landmarks.

[42] An image processing device comprising: a landmark detection unit that detects two or more joint landmarks that are landmarks related to the joint in a joint image that is an in-vivo image of a human joint; and a determination unit that determines a region of interest related to the joint in the joint image based on the two or more detected joint landmarks.

[43] The image processing device according to

[42] , further comprising a state detection unit that detects a joint state, which is a state of the joint, based on a local joint image that is an image within the determined region of interest among the joint images.

[0049] According to one aspect of the present disclosure, a more accurate region of interest can be determined.

[0050] 1 is a diagram illustrating an example of a system configuration of an image processing system according to an embodiment. FIG. 1 is a diagram illustrating an example of a functional configuration of a creation device according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer used in the creation device according to an embodiment. A flowchart illustrating an example of processing executed by the creation device according to an embodiment. A flowchart illustrating another example of processing executed by the creation device according to an embodiment. FIG. 2 is an example of a training ankle image created by the processing shown in FIG. 5. FIG. 3 is a diagram illustrating an example of a configuration of a creation program according to an embodiment. FIG. 4 is a diagram illustrating an example of a functional configuration of a learning device according to an embodiment. FIG. 5 is a diagram illustrating an example of a hardware configuration of a computer used in the learning device according to an embodiment. A diagram explaining an example of determining ROI1. A flowchart illustrating an example of processing executed by the learning device according to an embodiment. A flowchart illustrating another example of processing executed by the learning device according to an embodiment. FIG. 12 is an example of a training ankle image created by the processing shown in FIG. 12. FIG. 13 is a diagram illustrating a configuration of a learning program according to an embodiment. FIG. 14 is a diagram illustrating an example of a functional configuration of a determination device according to an embodiment. FIG. 15 is a diagram illustrating an example of a hardware configuration of a computer used in the determination device according to an embodiment. A flowchart illustrating an example of processing executed by the determination device according to an embodiment. A flowchart illustrating another example of processing executed by the determination device according to an embodiment. FIG. 16 is an example of a training ankle image created by the processing shown in FIG. 18. FIG. 17 is a diagram illustrating an example of joint landmarks in an ankle image. FIG. 18 is a diagram illustrating an example of joint landmarks in a knee image. FIG. 19 is a diagram illustrating another example of joint landmarks in a knee image. FIG. 20 is a diagram illustrating an example of joint landmarks in an elbow image. FIG. 10 is a diagram showing another example of joint landmarks in an elbow image. FIG. 11 is a diagram showing a configuration of a determination program according to an embodiment. FIG. 12 is a diagram showing an example of a functional configuration of a display device according to an embodiment. FIG. 13 is a diagram showing an example of a hardware configuration of a computer used in the display device according to an embodiment. FIG. 14 is a diagram showing a configuration of a display program according to an embodiment.

[0051] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicate explanations will be omitted. Furthermore, the embodiments of the present disclosure in the following description are specific examples of the present invention, and the present invention is not limited to these embodiments unless otherwise specified to limit the present invention.

[0052] FIG. 1 is a diagram showing an example of the system configuration of an image processing system 1 (image processing system) according to an embodiment. As shown in FIG. 1, the image processing system 1 is configured to include a creation device 2 according to an embodiment, a learning device 3 according to an embodiment, a determination device 4 (image processing device) according to an embodiment, and a display device 5 according to an embodiment. The image processing system 1 may further include any other device. The devices included in the image processing system 1 are communicatively connected to each other via a network such as a LAN (Local Area Network) or the Internet, and can send and receive information to and from each other. Two or more devices included in the image processing system 1 may be configured as a single device that has all the functions of the two or more devices.

[0053] [Creating Device 2] The creating device 2 is a computer device that creates training data. Fig. 2 is a diagram showing an example of the functional configuration of the creating device 2. As shown in Fig. 2, the creating device 2 includes a storage unit 20, an image reading unit 21, a landmark input unit 22, a joint state input unit 23, and an input correction unit 24.

[0054] Each functional block of the creation device 2 is assumed to function within the creation device 2, but this is not limited to this. For example, some of the functional blocks of the creation device 2 may function within a computer device different from the creation device 2 and connected to the creation device 2 via a network (e.g., the learning device 3, the determination device 4, or the display device 5), while appropriately transmitting and receiving information with the creation device 2. Furthermore, some functional blocks of the creation device 2 may be omitted, multiple functional blocks may be integrated into one functional block, or one functional block may be separated into multiple functional blocks.

[0055] FIG. 3 is a diagram illustrating an example of the hardware configuration of a computer used in the creation device 2. As shown in FIG. 3 , the creation device 2 is physically configured as a computer system including a CPU (Central Processing Unit) 200, which is a central processing unit (processor), a RAM (Random Access Memory) 201 and a ROM (Read Only Memory) 202, which are main storage devices, an input / output device 203 such as a keyboard, a microphone, or a display, a communication module 204, which is a data transmission / reception device, and an auxiliary storage device 205 such as a hard disk or an SSD (Solid State Drive). The CPU 200, the RAM 201, the ROM 202, the input / output device 203, the communication module 204, and the auxiliary storage device 205 may each be configured in plural. The functions of each functional block illustrated in FIG. 2 are realized by loading predetermined computer software onto hardware such as the CPU 200 and RAM 201 illustrated in FIG. 3 , which operates the input / output device 203 and the communication module 204 under the control of the CPU 200, and reads and writes data from and to the RAM 201 and the auxiliary storage device 205.

[0056] Hereinafter, each function of the creation device 2 shown in FIG. 2 will be described.

[0057] The storage unit 20 stores any information that is used or output during processing of the creation device 2. The storage unit 20 may store information calculated by each function of the creation device 2. The information stored by the storage unit 20 may be referenced as appropriate by each function of the creation device 2, or may be referenced as appropriate via a network by each function of the learning device 3, the determination device 4, or the display device 5.

[0058] The image reading unit 21 reads joint images, which are images of human joints (such as ankles, elbows, and knees) inside the body. The image reading unit 21 may read the joint images using the input / output device 203, or may read the joint images from another device or the like via a network using the communication module 204. For example, the image reading unit 21 may read the joint images from a joint image storage module (a module that stores joint images) of another device. When reading the joint images, the image reading unit 21 may adjust the image quality of the joint images, such as by enhancing the image quality (by adjusting gradation, removing noise, etc.). The image reading unit 21 may store the read joint images in the storage unit 20, or may output the read joint images to another functional block of the creation device 2.

[0059] In this embodiment, the "joint" may be a joint of a patient with a blood coagulation disorder or rheumatoid arthritis, and preferably a joint of a patient with hemophilia. In this embodiment, the "joint image" may be an ultrasound image or an X-ray image. In this embodiment, the "joint image" may be an image with enhanced image quality.

[0060] The landmark input unit 22 accepts input related to joint landmarks, which are landmarks related to joints, for a joint image. The landmark input unit 22 may accept input related to joint landmarks for a joint image read by the image reading unit 21. More specifically, the landmark input unit 22 may accept input related to joint landmarks for a joint image read by the image reading unit 21 and stored by the storage unit 20, or may accept input related to joint landmarks for a joint image read by the image reading unit 21 and output from the image reading unit 21 to the landmark input unit 22.

[0061] A joint landmark is, for example, a characteristic point (position, shape, pattern, brightness, brightness, etc.) on a joint image of an anatomical tissue organ of a joint. The joint landmark may be the tibia, talus, fibula, femur, patella, humerus, radius, ulna, cartilage, fat, joint capsule, or muscle. The appearance of joint landmarks, synovitis, and bleeding in joint images is disclosed, for example, in References 1 to 4 below. Reference 1: Journal of Kyorin Medical Association, Vol. 48, No. 1, pp. 67-73, March 2017 https: / / www.jstage.jst.go.jp / article / kyorinmed / 48 / 1 / 48_67 / _pdf Reference 2: "First Orthopedic Ultrasound Examination" https: / / www.konicaminolta.jp / healthcare / products / us / snible2 / pdf / snible_dr_minagawa.pdf Reference 3: Ultrasound Examination Techniques (1881-4506), Vol. 47, No. 2, pp. 153-157 (2022.04), DOI: 10.11272 / jss.r549 Reference 4: Pediatric Internal Medicine (0385-6305), Vol. 54, Supplement, pp. 671-676 (2022.12)

[0062] The landmark input unit 22 may receive input regarding joint landmarks from a user (e.g., a doctor) of the creation device 2. More specifically, the landmark input unit 22 may use the input / output device 203 to display a joint image to the user of the creation device 2 and may receive annotation and labeling of (one or more) joint landmarks by the user on the joint image. Annotation, for example, refers to drawing a line along a joint landmark or drawing a line to surround a joint landmark on the joint image. Labeling, for example, refers to associating each annotation line with a type of joint landmark (e.g., assigning a color according to the type of landmark). The landmark input unit 22 may receive annotation and labeling of two or more joint landmarks (including different labels for different joint landmarks) by the user of the creation device 2. As described above, the types of joint landmarks may be the tibia, talus, fibula, femur, patella, humerus, radius, ulna, cartilage, fat, joint capsule, or muscle.

[0063] The landmark input unit 22 may add landmark input information, which is information regarding the input related to the received joint landmarks, to the joint image, and then store the information in the storage unit 20 or output it to another functional block of the creation device 2.

[0064] The joint state input unit 23 receives input regarding the joint state, which is the state of the joint, for the joint image.

[0065] The joint state input unit 23 may accept input regarding the joint state for the joint image read by the image reading unit 21. More specifically, the joint state input unit 23 may accept input regarding the joint state for the joint image read by the image reading unit 21 and stored by the storage unit 20, or may accept input regarding the joint state for the joint image read by the image reading unit 21 and output from the image reading unit 21 to the joint state input unit 23.

[0066] The joint state input unit 23 may accept an input regarding the joint state for a joint image (e.g., a joint image annotated and labeled with joint landmarks) in which the landmark input information obtained by the landmark input unit 22 is reflected. More specifically, based on the joint image to which the landmark input information is added and stored by the storage unit 20, the joint state input unit 23 may reflect the landmark input information in the joint image and accept an input regarding the joint state for the reflected joint image, or based on the joint image to which the landmark input information is added and output from the landmark input unit 22 to the joint state input unit 23, the joint state input unit 23 may reflect the landmark input information in the joint image and accept an input regarding the reflected joint image.

[0067] The joint condition input unit 23 may indicate that the joint condition is normal, abnormal, bleeding, synovitis, or arthropathy. When indicating that the joint condition is abnormal, bleeding, synovitis, or arthropathy, the joint condition input unit 23 may further indicate the corresponding location in the joint image.

[0068] The joint state input unit 23 may receive input regarding the joint state from a user (such as a doctor) of the creation device 2. More specifically, the joint state input unit 23 may use the input / output device 203 to display a joint image (with or without the landmark input information reflected) to the user of the creation device 2, and may also receive labeling and annotation of the joint state by the user for the joint image. Labeling refers to, for example, attaching a normal label indicating that the joint image is normal, attaching an abnormal label indicating that the joint image is abnormal, or attaching an abnormal label indicating that the joint image is abnormal to an abnormal location. Annotation refers to, for example, drawing a line along or surrounding the abnormal location on a joint image containing an abnormal location or a joint image labeled with an abnormality. The joint state input unit 23 may receive annotations of two or more joint states by the user of the creation device 2.

[0069] The joint state input unit 23 may add joint state input information, which is information regarding the input regarding the received joint state, to the joint image and then store it in the storage unit 20 or output it to other functional blocks of the creation device 2.

[0070] The input correction unit 24 accepts corrections to the input related to the joint landmarks for the joint image and corrections to the input related to the joint state. More specifically, the input correction unit 24 accepts corrections to the input related to the joint landmarks for the joint image in which the landmark input information is reflected, and accepts corrections to the input related to the joint state for the joint image in which the joint state input information is reflected. The input correction unit 24 may accept corrections to the input from a user different from the user of the creation device 2 who made the input related to the joint landmarks. The input correction unit 24 may accept corrections to the input from a user different from the user of the creation device 2 who made the input related to the joint state.

[0071] The input correction unit 24 may add the corrected landmark input information to the joint image, and then store it in the storage unit 20, or output it to another functional block of the creation device 2. The input correction unit 24 may add the corrected joint state input information to the joint image, and then store it in the storage unit 20, or output it to another functional block of the creation device 2.

[0072] The joint images handled by the creating device 2 are data (teaching data) that are ultimately used for learning by the learning device 3. Therefore, the joint images handled by the creating device 2 are appropriately referred to as learning joint images.

[0073] Fig. 4 is a flowchart showing an example of processing executed by the creating device 2. More specifically, Fig. 4 is a flowchart showing processing for adding landmark input information and joint state input information to a learning joint image.

[0074] First, the image reading unit 21 reads a training joint image (step S200). Next, user A, who is a user of the creation device 2, determines whether all the landmarks necessary for the joint are present in the training joint image (displayed by the creation device 2) (step S201). If it is determined that all the landmarks are present in step S201 (S201: YES), user A annotates (including labels) all the necessary landmarks for the joint in the training joint image, and the landmark input unit 22 accepts this input (step S202). Next, user A determines whether there is an abnormality in the joint in the training joint image (step S203).

[0075] If it is determined in step S203 that an abnormality exists (S203: YES), user A annotates the abnormality in the joint in the training joint image, and the joint state input unit 23 accepts the input (step S204). Next, user A labels the training joint image as abnormal, and the joint state input unit 23 accepts the input (step S205). On the other hand, if it is determined in step S203 that no abnormality exists (S203: NO), user A labels the training joint image as normal, and the joint state input unit 23 accepts the input (step S206).

[0076] Following step S205 or step S206, user B, who is a user of the creation device 2 different from user A, performs a secondary judgment to judge the input by user A up to that point, and judges whether the secondary judgment is successful or not (step S207). If it is judged in step S207 that the input is not successful (S207: NO), user B corrects the incorrect parts of the annotations and / or labels by user A up to that point, and the input correction unit 24 accepts the corrections (step S208).

[0077] On the other hand, if it is determined in step S201 that all the training joint images are not complete (S201: NO), user A removes the training joint images loaded in step S200 from the training joint images (step S209). Following step S208 or step S209, or if the determination in step S207 is successful (S207: YES), the creation device 2 determines whether or not the loading of all the training joint images is complete (step S210). If it is determined in step S210 that the loading is complete (S210: YES), the process ends. On the other hand, if it is determined in step S210 that the loading is not complete (S210: NO), the process from step S200 is repeated for the remaining training joint images.

[0078] Fig. 5 is a flowchart showing another example of processing executed by the creation device 2. More specifically, Fig. 5 is a concrete example of determining bleeding in the ankle in accordance with the flowchart shown in Fig. 4. For example, the training joint image in Fig. 4 is replaced with a training ankle image specialized for the ankle in Fig. 5. When the joint is the ankle, the tibia, talus, and fat may be used as landmarks.

[0079] First, the image reading unit 21 reads a training ankle image (step S200a). Next, user A, who is a user of the creation device 2, determines whether the training ankle image (displayed by the creation device 2) contains the necessary tibia, talus, and fat for the ankle (step S201a). If it is determined that they are present in step S201a (YES in S201a), user A annotates (including labels) all of the tibia, talus, and fat in the training ankle image, and the landmark input unit 22 accepts this input (step S202a). Next, user A determines whether there is any abnormality in the ankle in the training ankle image (step S203a).

[0080] If it is determined in step S203a that an abnormality exists (S203a: YES), user A annotates the ankle abnormality in the training ankle image, and the joint state input unit 23 accepts the input (step S204a). Next, user A labels the training ankle image as abnormal, and the joint state input unit 23 accepts the input (step S205a). On the other hand, if it is determined in step S203a that no abnormality exists (S203a: NO), user A labels the training ankle image as normal, and the joint state input unit 23 accepts the input (step S206a).

[0081] Following step S205a or step S206a, user B, who is a user of the creation device 2 different from user A, performs a secondary judgment to judge the input by user A up to that point, and judges whether the secondary judgment is successful or not (step S207a). If it is judged in step S207a that the input is not successful (S207a: NO), user B corrects the erroneous parts of the annotations and / or labels by user A up to that point, and the input correction unit 24 accepts the corrections (step S208a).

[0082] On the other hand, if it is determined in step S201a that all the training ankle images are not complete (S201a: NO), user A removes the training ankle image read in step S200a from the training ankle images (step S209a). Following step S208a or step S209a, or if the result of step S207a is "pass" (S207a: YES), the creation device 2 determines whether or not the reading of all the training ankle images is complete (step S210a). If it is determined in step S210a that the reading is complete (S210a: YES), the process ends. On the other hand, if it is determined in step S210a that the reading is not complete (S210a: NO), the process from step S200a is repeated for the remaining training ankle images.

[0083] Fig. 6 is an example of a training ankle image created by the process shown in Fig. 5. The training ankle image shown in Fig. 6 shows the tibia, talus, and fat annotated (including labeling) in step S202a in Fig. 5, as well as the bleeding annotated in step S204a in Fig. 5.

[0084] Hereinafter, a training joint image to which at least one of landmark input information and joint state input information has been added by the creation device 2 will be simply referred to as a training joint image, as appropriate.

[0085] Next, a description will be given of a creation program P2 that causes a computer to execute a series of processes by the creation device 2. The creation program P2 is stored in a program storage area formed in the auxiliary storage device 205 included in the creation device 2, for example, as shown in FIG.

[0086] The creation program P2 is configured to include a storage module P20, an image reading module P21, a landmark input module P22, a joint state input module P23, and an input correction module P24. Functions realized by executing the storage module P20, the image reading module P21, the landmark input module P22, the joint state input module P23, and the input correction module P24 are similar to the functions of the storage unit 20, the image reading unit 21, the landmark input unit 22, the joint state input unit 23, and the input correction unit 24 of the creation device 2 described above. The creation program P2 is a program that causes (one or more CPUs of) the creation device 2 to function as the storage unit 20, the image reading unit 21, the landmark input unit 22, the joint state input unit 23, and the input correction unit 24.

[0087] The creation program P2 may be configured so that part or all of it is transmitted via a transmission medium such as a communication line, and is received and stored (including installed) by another device. Furthermore, each module of the creation program P2 may be installed on one of multiple computers, rather than on a single computer. In this case, the series of processes of the creation program P2 described above are performed by a computer system consisting of the multiple computers.

[0088] [Learning Device 3] The learning device 3 is a computer device that learns various detection models based on the training joint images that are the teacher data created by the creation device 2. Fig. 8 is a diagram showing an example of the functional configuration of the learning device 3. As shown in Fig. 8, the learning device 3 includes a storage unit 30, an image reading unit 31, an image quality processing unit 32, a landmark detection model learning unit 33, an ROI1 determination unit 34, and a state detection model learning unit 35.

[0089] Each functional block of the learning device 3 is assumed to function within the learning device 3, but this is not limited to this. For example, some of the functional blocks of the learning device 3 may function within a computer device different from the learning device 3 and connected to the learning device 3 via a network (e.g., the creation device 2, the determination device 4, or the display device 5), while appropriately sending and receiving information with the learning device 3. Furthermore, some functional blocks of the learning device 3 may be omitted, multiple functional blocks may be integrated into one functional block, or one functional block may be broken down into multiple functional blocks.

[0090] FIG. 9 is a diagram showing an example of the hardware configuration of a computer used in the learning device 3. As shown in FIG. 9 , the learning device 3 is physically configured as a computer system including a CPU 300, a RAM 301, a ROM 302, an input / output device 303 such as a keyboard, a microphone, or a display, a communication module 304 which is a data transmission / reception device, and an auxiliary storage device 305 such as a hard disk or SSD. The CPU 300, the RAM 301, the ROM 302, the input / output device 303, the communication module 304, and the auxiliary storage device 305 may each be configured in plural. The functions of each functional block shown in FIG. 8 are realized by loading predetermined computer software onto hardware such as the CPU 300 and RAM 301 shown in FIG. 9 , operating the input / output device 303 and the communication module 304 under the control of the CPU 300, and reading and writing data from and to the RAM 301 and the auxiliary storage device 305.

[0091] Hereinafter, each function of the learning device 3 shown in FIG. 8 will be described.

[0092] The storage unit 30 stores any information that is used or output during processing of the learning device 3. The storage unit 30 may store information calculated by each function of the learning device 3. The information stored by the storage unit 30 may be referenced as appropriate by each function of the learning device 3, or may be referenced as appropriate via a network by each function of the creation device 2, the determination device 4, or the display device 5.

[0093] The image reading unit 31 reads a learning joint image. The image reading unit 31 may read the learning joint image using the input / output device 303, or may read the learning joint image from another device or the like via a network using the communication module 304. For example, the image reading unit 31 may read a learning joint image (to which landmark input information and joint state input information have been added by the creation device 2) from the storage unit 20 of the creation device 2. The image reading unit 31 may store the read learning joint image in the storage unit 30, or may output the read learning joint image to another functional block of the learning device 3.

[0094] The image quality processing unit 32 adjusts the image quality of the learning joint images, such as by enhancing the image quality of the learning joint images (by adjusting gradation, removing noise, etc.). The image quality processing unit 32 may adjust the image quality of the learning joint images read by the image reading unit 31. More specifically, the image quality processing unit 32 may adjust the image quality of the learning joint images read by the image reading unit 31 and stored by the storage unit 30, or may adjust the image quality of the learning joint images read by the image reading unit 31 and output from the image reading unit 31 to the image quality processing unit 32. The image quality processing unit 32 may store the learning joint images whose image quality has been adjusted in the storage unit 30, or may output them to another functional block of the learning device 3.

[0095] The landmark detection model learning unit 33 performs learning using the training joint images to generate a landmark detection model. The landmark detection model learning unit 33 may perform learning using the training joint images read by the image reading unit 31, or may perform learning using training joint images whose image quality has been adjusted by the image quality processing unit 32. More specifically, the landmark detection model learning unit 33 may perform learning using the training joint images read by the image reading unit 31 and stored by the storage unit 30, or may perform learning using the training joint images read by the image reading unit 31 and output from the image reading unit 31 to the landmark detection model learning unit 33, or may perform learning using the training joint images whose image quality has been adjusted by the image quality processing unit 32 and stored by the storage unit 30, or may perform learning using the training joint images whose image quality has been adjusted by the image quality processing unit 32 and output from the image quality processing unit 32 to the landmark detection model learning unit 33.

[0096] The landmark detection model is a trained model that detects joint landmarks in a joint image when the joint image is input. The landmark detection model may be a model trained based on training data consisting of pairs of joint images and images in which joint landmarks in the joint images are annotated. For example, the landmark detection model training unit 33 performs training based on training data consisting of pairs of training joint images that do not reflect landmark input information and training joint images that reflect the landmark input information, and generates a landmark detection model.

[0097] The landmark detection model may be a trained model that, when a joint image is input, detects two or more joint landmarks in the joint image. The landmark detection model may be a model trained based on training data consisting of pairs of a joint image and an image in which two or more joint landmarks are annotated in the joint image. For example, the landmark detection model training unit 33 performs training based on training data consisting of pairs of training joint images that do not reflect landmark input information that includes annotations of two or more joint landmarks, and training joint images that reflect the landmark input information (including annotations of two or more joint landmarks), thereby generating a landmark detection model.

[0098] The landmark detection model learning unit 33 may learn a landmark detection model for each joint.

[0099] The ROI1 determination unit 34 detects joint landmarks in the training joint images and determines (detects, identifies, cuts out) ROI1, which is a Region of Interest (ROI), based on the positions of the detected joint landmarks. The ROI1 determination unit 34 may detect joint landmarks based on landmark input information added to the training joint images, or may detect joint landmarks using a landmark detection model generated by the landmark detection model learning unit 33. The ROI1 determination unit 34 may determine ROI1 based on two or more detected joint landmarks. The ROI1 determination unit 34 may also determine ROI1 based on the type of each detected joint landmark.

[0100] The ROI1 determination unit 34 may determine ROI1 based on the offset of the detected joint landmark. More specifically, the ROI1 determination unit 34 may determine a surrounding offset based on the label and relative position of the detected joint landmark, and then determine ROI1. The ROI1 determination unit 34 may determine ROI1 based on the distance from the center of the detected joint landmark to the boundary line of ROI1.

[0101] FIG. 10 is a diagram illustrating an example of determining ROI1. As shown in FIG. 10 , the tibia, talus, and fat at the ankle are used as examples of joint landmarks. A method for determining ROI1 (ankle joint ROI) using the labels and relative positions of two or more joint landmarks will be described. First, node numbers 1, 2, and 3 are assigned to the centers (the midpoints of the lines or the geometric centers) of the tibia, fat, and talus, respectively (hereinafter referred to as N1, N2, and N3). Next, the distances (L4, L7, and L8) from each node (joint landmark) to other nodes are measured to determine the relative positional relationships between the joint landmarks. Next, the offset distance from each node (joint landmark) is measured around the circumference, and the boundaries (L1, L5, L11, and L10) of ROI1 are determined at the top, bottom, left, and right positions on the coordinate system, thereby determining the ROI. The more joint landmarks there are, the more accurate the estimation of the relative positions and the determination of ROI1 become.

[0102] The offset distance refers to the distance between the center of a joint landmark (N1, N2, and N3 in FIG. 10 ) and the boundary of ROI1 (e.g., L1, L5, L11, and L10 in FIG. 10 ), as well as the distance between joint landmarks (e.g., L4, L7, and L8). The specific value of the offset distance may be determined from a clinical (human joint structure) perspective. For example, the boundary of ROI1 in FIG. 10 can be determined by setting the offset distance from the center of each joint landmark (N1, N2, and N3 in FIG. 10 ) so that the joints are properly displayed within ROI1 (so that joint abnormalities and joint landmarks are displayed). For example, in FIG. 10 , the boundary of ROI1 is determined based on the offset distance to the left from N1, which is located at the leftmost position, the offset distance upward from N2, which is located at the top, and the offset distance to the right and downward from N3, which is located at the rightmost and bottom. From a clinical perspective, the offset distance may be set as a distance that prevents the joint area from appearing further outside ROI1. The offset distance may vary depending on whether the subject is an adult or a child, male or female, or race (Asian or Western). For example, the size of the joint ROI1 is smaller for children than for adults, for women than for men, and for Asians than for Westerners, so the corresponding offset distance may be reduced accordingly. When the offset distance varies, the offset distances for adults, children, men, women, and races can be artificially input into the learning device 3 or the determination device 4 to extract an ROI1 corresponding to the offset distance for each subject. An ROI1 corresponding to the offset distance for each subject may be extracted using a trained model trained based on joint images in which the offset distances of the joint images are labeled for adults, children, men, women, and races.

[0103] As described above, ROI1 is determined based on the labels and relative positions of joint landmarks. The type of joint landmark is identified by its label. For example, the left boundary of ROI1 is determined by the L1 offset distance of the "tibia" (identified by its label). In this way, knowing the labels of the joint landmarks makes it possible to identify, for example, the left boundary of ROI1. The relative positions can be used to estimate some joint landmarks when they are not detected. In FIG. 10 , the tibia, fat, and talus of the ankle are detected. If any of these remains undetected, the undetected joint landmark can be estimated based on the labels (types) and relative positions of the detected joint landmarks. For example, if the talus remains undetected, it can be estimated that the intersection of two circles at a distance L8 from the tibia and a distance L7 from the fat is the talus. In this way, the undetected joint landmark can be estimated based on its relative position. This allows for a more accurate determination of the range of ROI1.

[0104] The ROI 1 determination unit 34 may store ROI 1 information, which is information relating to the determined ROI 1, in the storage unit 30 or may output it to other functional blocks of the learning device 3.

[0105] The condition detection model learning unit 35 performs learning using local joint images, which are images within the ROI1 determined by the ROI1 determination unit 34, among the learning joint images, to generate a condition detection model. The learning joint images are the same as the learning joint images used by the landmark detection model learning unit 33. The local joint images may be images within the ROI1 indicated by the ROI1 information stored by the storage unit 30, among the learning joint images, or may be images within the ROI1 indicated by the ROI1 information output from the ROI1 determination unit 34 to the condition detection model learning unit 35.

[0106] The condition detection model is a trained model that, when a local joint image is input, detects the joint state of the joint indicated by the local joint image. The condition detection model may be a model trained based on training data consisting of pairs of a local joint image and the local joint image associated with a normal label or a local joint image associated with an abnormal location of the joint indicated by the local joint image with an abnormal label. For example, the condition detection model training unit 35 performs training based on training data consisting of pairs of local joint images of training joint images that do not reflect the joint state input information and local joint images of training joint images that reflect the joint state input information, thereby generating the condition detection model.

[0107] Fig. 11 is a flowchart showing an example of processing executed by the learning device 3. More specifically, Fig. 11 is a flowchart showing processing for learning a landmark detection model and a state detection model based on learning joint images.

[0108] First, the image reading unit 31 reads the training joint image (step S300). Next, the image quality processing unit 32 adjusts the image quality of the training joint image read in step S300 (step S301). Note that step S301 may be omitted. Next, the landmark detection model learning unit 33 performs learning using the training joint image whose image quality has been adjusted in step S301 (if step S301 is omitted, the training joint image read in step S300; the same applies below) to generate a landmark detection model (step S302). Next, the ROI1 determination unit 34 detects joint landmarks in the training joint image whose image quality has been adjusted in step S301 (step S303), and cuts out (determines) ROI1 based on the positions of the detected joint landmarks (step S304). Next, the state detection model learning unit 35 performs learning using the local joint image within ROI1 extracted in step S304 from the training joint images whose image quality has been adjusted in step S301, and generates a state detection model (step S305).

[0109] Fig. 12 is a flowchart showing another example of processing executed by the learning device 3. More specifically, Fig. 12 is a concrete example of the flowchart shown in Fig. 11 , in which various detection models for determining bleeding in the ankle are learned. For example, the training joint image in Fig. 11 is replaced with a training ankle image specialized for the ankle in Fig. 12. When the joint is the ankle, the tibia, talus, and fat may be used as landmarks.

[0110] First, the image reading unit 31 reads a training ankle image (step S300a). Next, the image quality processing unit 32 adjusts the image quality of the training ankle image read in step S300a (step S301a). Note that step S301a may be omitted. Next, the landmark detection model training unit 33 performs training using the training ankle image whose image quality has been adjusted in step S301a (or the training ankle image read in step S300a if step S301a is omitted; the same applies below) to generate a landmark detection model (step S302a). Next, the ROI1 determination unit 34 detects the tibia, talus, and fat in the training ankle image whose image quality has been adjusted in step S301a (step S303a), and cuts out (determines) ROI1 (ankle local ROI) based on the positions of the detected tibia, talus, and fat (step S304a). Next, the condition detection model learning unit 35 performs learning using the local joint image within ROI1 extracted in step S304a from the training ankle image whose image quality has been adjusted in step S301a, and generates a condition detection model (step S305a).

[0111] Fig. 13 is an example of a learning ankle image created by the process shown in Fig. 12. The learning ankle image shown in Fig. 13 shows ROI1 cut out in step S304a of Fig. 12.

[0112] Next, we will explain the learning program P3 that causes a computer to execute a series of processes by the learning device 3. The learning program P3 is stored in a program storage area formed in the auxiliary storage device 305 provided in the learning device 3, for example, as shown in Figure 14.

[0113] The learning program P3 is configured to include a storage module P30, an image reading module P31, an image quality processing module P32, a landmark detection model learning module P33, an ROI1 determination module P34, and a state detection model learning module P35. Functions realized by executing the storage module P30, the image reading module P31, the image quality processing module P32, the landmark detection model learning module P33, the ROI1 determination module P34, and the state detection model learning module P35 are similar to the functions of the storage unit 30, the image reading unit 31, the image quality processing module 32, the landmark detection model learning unit 33, the ROI1 determination unit 34, and the state detection model learning unit 35 of the learning device 3 described above. The learning program P3 is a program for causing (one or more CPUs of) the learning device 3 to function as the storage unit 30, the image reading unit 31, the image quality processing module 32, the landmark detection model learning unit 33, the ROI1 determination unit 34, and the state detection model learning unit 35.

[0114] The learning device 3 may be configured so that part or all of it is transmitted via a transmission medium such as a communication line, and is received and stored (including installed) by another device. Furthermore, each module of the learning device 3 may be installed on one of multiple computers, rather than on a single computer. In this case, the series of processes of the learning device 3 described above are performed by a computer system consisting of the multiple computers.

[0115] [Determination Device 4] The determination device 4 is a computer device that makes various determinations based on various detection models learned by the learning device 3. The determination device 4 (or the image processing system 1) may be a computer device that supports diagnosis of a joint condition based on a joint image. FIG. 15 is a diagram showing an example of the functional configuration of the determination device 4. As shown in FIG. 15, the determination device 4 includes a storage unit 40, an image reading unit 41, an image quality processing unit 42, a landmark detection unit 43, an ROI1 determination unit 44, a state detection unit 45, an ROI2 determination unit 46, and a confidence factor visualization unit 47.

[0116] Each functional block of the determination device 4 is assumed to function within the determination device 4, but is not limited to this. For example, some of the functional blocks of the determination device 4 may function within a computer device different from the determination device 4 and connected to the determination device 4 via a network (e.g., the creation device 2, the learning device 3, or the display device 5), while appropriately transmitting and receiving information with the determination device 4. Furthermore, some functional blocks of the determination device 4 may be omitted, multiple functional blocks may be integrated into one functional block, or one functional block may be separated into multiple functional blocks.

[0117] FIG. 16 is a diagram showing an example of the hardware configuration of a computer used in the determination device 4. As shown in FIG. 16 , the determination device 4 is physically configured as a computer system including a CPU 400, a RAM 401, a ROM 402, an input / output device 403 such as a keyboard, a microphone, or a display, a communication module 404 which is a data transmission / reception device, and an auxiliary storage device 405 such as a hard disk or SSD. The CPU 400, the RAM 401, the ROM 402, the input / output device 403, the communication module 404, and the auxiliary storage device 405 may each be configured in plural. The functions of each functional block shown in FIG. 15 are realized by loading predetermined computer software onto hardware such as the CPU 400 and RAM 401 shown in FIG. 16 , thereby operating the input / output device 403 and the communication module 404 under the control of the CPU 400 and reading and writing data from and to the RAM 401 and the auxiliary storage device 405.

[0118] Hereinafter, each function of the determination device 4 shown in FIG. 15 will be described.

[0119] The storage unit 40 stores any information that is used or output in the processing of the determination device 4. The storage unit 40 may store information calculated by each function of the determination device 4. The information stored by the storage unit 40 may be referenced as appropriate by each function of the determination device 4, or may be referenced as appropriate via a network by each function of the creation device 2, the learning device 3, or the display device 5.

[0120] The image reading unit 41 reads a joint image to be determined, which is an image of a joint to be determined by the determination device 4. The image reading unit 41 may read the joint image to be determined using the input / output device 403, or may read the joint image to be determined from another device or the like via a network using the communication module 404. For example, the image reading unit 41 may read the joint image to be determined from another device, such as a joint image capturing device (a device that captures joint images). The image reading unit 41 may store the read joint image to be determined in the storage unit 40, or may output the read joint image to another functional block of the determination device 4.

[0121] The image quality processing unit 42 adjusts the image quality of the to-be-judged joint image, such as by enhancing the image quality (by adjusting gradation, removing noise, etc.) of the to-be-judged joint image. The image quality processing unit 42 may adjust the image quality of the to-be-judged joint image read by the image reading unit 41. More specifically, the image quality processing unit 42 may adjust the image quality of the to-be-judged joint image read by the image reading unit 41 and stored by the storage unit 40, or may adjust the image quality of the to-be-judged joint image read by the image reading unit 41 and output from the image reading unit 41 to the image quality processing unit 42. The image quality processing unit 42 may store the to-be-judged joint image, after image quality adjustment, in the storage unit 40, or may output it to another functional block of the judgment device 4.

[0122] The landmark detection unit 43 detects joint landmarks in the to-be-determined joint images. The landmark detection unit 43 may detect joint landmarks in the to-be-determined joint images read by the image reading unit 41, or may detect joint landmarks in the to-be-determined joint images whose image quality has been adjusted by the image quality processing unit 42. More specifically, the landmark detection unit 43 may detect joint landmarks in the to-be-determined joint images read by the image reading unit 41 and stored by the storage unit 40, or may detect joint landmarks in the to-be-determined joint images read by the image reading unit 41 and output from the image reading unit 41 to the landmark detection unit 43, or may detect joint landmarks in the to-be-determined joint images whose image quality has been adjusted by the image quality processing unit 42 and stored by the storage unit 40, or may detect joint landmarks in the to-be-determined joint images whose image quality has been adjusted by the image quality processing unit 42 and output from the image quality processing unit 42 to the landmark detection unit 43.

[0123] The landmark detection unit 43 may detect two or more joint landmarks from the joint image to be determined. One of the two or more detected joint landmarks may be a bone, and the rest may be a bone other than the bone, cartilage, fat, joint capsule, or muscle. The landmark detection unit 43 may perform the detection using a landmark detection model that detects two or more joint landmarks in a joint image when the joint image is input. The landmark detection unit 43 may also detect the type of each joint landmark.

[0124] When the landmark detection unit 43 detects a joint landmark, if the joint landmark cannot be detected or the degree of certainty is low, the positions of other landmarks may be estimated or corrected based on the relative positions of the learned joint landmarks.

[0125] The landmark detection unit 43 may add landmark detection information, which is information about the detected joint landmarks, to the to-be-judged joint image, and then store the information in the storage unit 40 or output the information to another functional block of the judgment device 4.

[0126] The ROI1 determination unit 44 determines ROI1, which is a region of interest related to a joint, in the joint image to be determined, based on the joint landmarks detected by the landmark detection unit 43. More specifically, the ROI1 is determined in the joint image to be determined based on the joint landmarks indicated by the landmark detection information output from the landmark detection unit 43 to the ROI1 determination unit 44. The ROI1 determination unit 44 may determine ROI1 in the joint image to be determined based on two or more joint landmarks detected by the landmark detection unit 43. The ROI1 determination unit 44 may also determine ROI1 based on the type of each joint landmark detected by the landmark detection unit 43. The ROI1 determination unit 44 may determine ROI1 based on the distance from the center of the joint landmark detected by the landmark detection unit 43 (each of the two or more joint landmarks detected by the landmark detection unit 43) to the boundary line of ROI1. The method of determining ROI1 by the ROI1 determination unit 44 is similar to the method of determining ROI1 by the ROI1 determination unit 34 described above.

[0127] The ROI 1 determination unit 44 may store ROI 1 information, which is information relating to the determined ROI 1, in the storage unit 40 or may output the information to another functional block of the determination device 4 .

[0128] The state detection unit 45 detects the joint state based on a local joint image, which is an image within the ROI1 determined by the ROI1 determination unit 44, among the joint images to be determined. The joint image to be determined is the same as the joint image to be determined used by the landmark detection unit 43. The local joint image may be an image within the ROI1 indicated by the ROI1 information stored by the storage unit 40, among the joint images to be determined, or an image within the ROI1 indicated by the ROI1 information output from the ROI1 determination unit 44 to the state detection unit 45. The state detection unit 45 may detect the joint state using a state detection model that, when a local joint image is input, detects the joint state of the joint indicated by the local joint image.

[0129] The state detection unit 45 may add joint state detection information, which is information regarding the detected joint state, to the local joint image or the joint image to be judged, and then store it in the storage unit 40 or output it to another functional block of the judgment device 4.

[0130] If the joint condition is abnormal, bleeding, synovitis, or arthropathy, the condition detection unit 45 may further indicate the corresponding location in the local joint image.

[0131] The joint may be an ankle joint, the joint landmark may be a tibia, fibula, talus, or fat, and the joint status may indicate normal, abnormal, bleeding, synovitis, or arthrosis. The joint may be a knee joint, the joint landmark may be a femur, tibia, fibula, patella, or fat, and the joint status may indicate normal, abnormal, bleeding, synovitis, or arthrosis. The joint may be an elbow joint, the joint landmark may be a humerus, radius, ulna, or fat, and the joint status may indicate normal, abnormal, bleeding, synovitis, or arthrosis.

[0132] The ROI2 determination unit 46 determines (cuts out, detects, identifies) an ROI2, which is a region of interest including an abnormal location, in the local joint image when a joint state indicating an abnormality is detected by the state detection unit 45. For example, when the joint state detection information output from the state detection unit 45 to the ROI2 determination unit 46 indicates that a joint state indicating an abnormality has been detected, the ROI2 determination unit 46 determines, as the ROI2, a rectangular region including the location in the local joint image indicated by the joint state.

[0133] The ROI 2 determination unit 46 may store ROI 2 information, which is information relating to the determined ROI 2, in the storage unit 40 or may output the information to another functional block of the determination device 4.

[0134] The confidence visualization unit 47 visualizes the portion of the local joint image that influenced the state detection model when it detected the joint state in a display format corresponding to the degree of influence (determination basis, confidence, determination confidence). The visualization by the confidence visualization unit 47 may use the existing technology Grad-CAM algorithm. As a display format, the confidence visualization unit 47 may display a heat map, or may display normal areas in blue and abnormal areas in red. The confidence visualization unit 47 may instruct a result display unit 51 of the display device 5 (described later) to perform visualization, and it may be the result display unit 51 that actually performs the visualization.

[0135] Fig. 17 is a flowchart showing an example of a process (image processing method) executed by the determination device 4. More specifically, Fig. 17 is a flowchart showing a process for determining whether or not there is an abnormality in the joint in the image data of the joint to be determined.

[0136] First, the image reading unit 41 reads the joint image to be determined (step S400). Next, the image quality processing unit 42 adjusts the image quality of the joint image to be determined read in step S400 (step S401). Note that step S401 may be omitted. Next, the landmark detection unit 43 detects joint landmarks from the joint image to be determined whose image quality has been adjusted in step S401 (or the joint image to be determined read in step S400 if step S401 is omitted; the same applies below) (step S402). Next, the determination device 4 determines whether landmarks necessary for the joint have been detected in step S402 (step S403). For example, it determines whether ROI1 can be determined based on at least one of the type and number of landmarks detected in step S402. For example, if two or more landmarks can estimate the boundary line of ROI1 (see FIG. 10), ROI1 can be determined. If the boundary line cannot be estimated, ROI1 cannot be determined.

[0137] If it is determined in step S403 that a joint landmark has been detected (S403: YES), the ROI1 determination unit 44 determines ROI1 in the to-be-judged joint image, the image quality of which has been adjusted in step S401, based on the joint landmark detected in step S402 (step S404). Next, the condition detection unit 45 detects the joint condition (presence or absence of abnormality) based on the local joint image, which is an image within ROI1 determined in step S404, among the to-be-judged joint images, the image quality of which has been adjusted in step S401 (step S405), and determines whether or not there is an abnormality in the joint (step S406).

[0138] If it is determined in step S406 that an abnormality exists (S406: YES), it is determined that an abnormal joint is included in the local joint image (step S407). Next, the ROI2 determination unit 46 determines ROI2, which is a region of interest including the abnormal location in the local joint image (step S408). On the other hand, if it is determined in step S406 that no abnormality exists (S406: NO), it is determined that a normal joint is included in the local joint image (no abnormal joint is included) (step S409). Following step S408 or step S409, the certainty visualization unit 47 visualizes the determination certainty (step S410).

[0139] On the other hand, if it is determined in step S403 that no detection has been made (S403: NO), the determination is unsuccessful (step S411). Following step S410 or step S411, the determination device 4 determines whether or not reading of all of the joint images to be determined has been completed (step S412). If it is determined in step S412 that reading has been completed (S412: YES), the process ends. On the other hand, if it is determined in step S412 that reading has not been completed (S412: NO), the process from step S400 is repeated for the remaining joint images to be determined.

[0140] Fig. 18 is a flowchart showing another example of processing executed by the determination device 4. More specifically, Fig. 18 is a concrete example of determining bleeding in the ankle in accordance with the flowchart shown in Fig. 17. For example, the joint image to be determined in Fig. 17 is replaced with an ankle image to be determined that is specialized for the ankle in Fig. 18. When the joint is the ankle, the tibia, talus, and fat may be used as landmarks.

[0141] First, the image reading unit 41 reads the ankle image to be judged (step S400a). Next, the image quality processing unit 42 adjusts the image quality of the ankle image to be judged read in step S400a (step S401a). Note that step S401a may be omitted. Next, the landmark detection unit 43 detects joint landmarks from the ankle image to be judged whose image quality has been adjusted in step S401a (if step S401a is omitted, the ankle image to be judged read in step S400a; the same applies below) (step S402a). Next, the judgment device 4 judges whether or not landmarks necessary for identifying the ankle have been detected in the detection in step S402a (step S403a).

[0142] If it is determined in step S403a that a joint landmark has been detected (S403a: YES), the ROI1 determination unit 44 determines ROI1 in the to-be-judged ankle image whose image quality has been adjusted in step S401a based on the joint landmark detected in step S402a (step S404a). Next, the condition detection unit 45 detects the joint condition (presence or absence of abnormality) based on the local joint image, which is an image within ROI1 determined in step S404a, among the to-be-judged ankle image whose image quality has been adjusted in step S401a (step S405a), and determines whether or not there is an abnormality in the ankle (step S406a).

[0143] If it is determined in step S406a that an abnormality exists (S406a: YES), it is determined that an abnormal ankle is included in the local joint image (step S407a). Next, the ROI2 determination unit 46 determines ROI2, which is a region of interest including the abnormality, in the local joint image (step S408a). On the other hand, if it is determined in step S406a that no abnormality exists (S406a: NO), it is determined that a normal ankle is included in the local joint image (no abnormal ankle is included) (step S409a). Following step S408a or step S409a, the certainty visualization unit 47 visualizes the determination certainty (step S410a).

[0144] On the other hand, if it is determined in step S403a that no detection has been made (S403a: NO), the determination is unsuccessful (step S411a). Following step S410a or step S411a, the determination device 4 determines whether or not reading of all of the ankle images to be determined has been completed (step S412a). If it is determined in step S412a that reading has been completed (S412a: YES), the process ends. On the other hand, if it is determined in step S412a that reading has not been completed (S412a: NO), the process from step S400a is repeated for the remaining ankle images to be determined.

[0145] Fig. 19 is an example of a training ankle image created by the process shown in Fig. 18. The training ankle image shown in Fig. 19 shows ROI1 extracted in step S404a of Fig. 18, ROI2 extracted in step S408a, and the determination certainty visualized in step S410a (a determination certainty visualized heat map). The determination certainty indicates the influence of the characteristics of the region with respect to abnormality (bleeding) on ​​the determination result as "high (high possibility of joint abnormality (bleeding or synovitis))", "medium", "low", or "even lower" depending on the shading of the region.

[0146] Examples of joint landmarks in a joint image will be described with reference to FIGS. 20 to 24.

[0147] Fig. 20 is a diagram showing an example of joint landmarks in an ankle image, in which the joint capsule, tibia, fat, cartilage, and talus are shown as joint landmarks in the ankle image.

[0148] Fig. 21 is a diagram showing an example of joint landmarks in a knee image, in which the femur, patella, fat, and other landmarks are shown as joint landmarks in the knee image.

[0149] 22 is a diagram showing another example of joint landmarks in a knee image, in which the quadriceps, fat, femur, and patella are shown as joint landmarks in the knee image.

[0150] Fig. 23 is a diagram showing an example of joint landmarks of an elbow image, in which the capitellum of the humerus, radius, fat, and other landmarks are shown as joint landmarks of the elbow image.

[0151] 24 is a diagram showing another example of joint landmarks of an elbow image, in which the brachioradialis muscle, joint capsule, fat, cartilage, capitellum, and radius are shown as joint landmarks of the elbow image.

[0152] Next, a description will be given of a determination program P4 (image processing program) for causing a computer to execute a series of processes by the determination device 4. The determination program P4 is stored in a program storage area formed in the auxiliary storage device 405 provided in the determination device 4, for example, as shown in FIG.

[0153] The determination program P4 is configured to include a storage module P40, an image reading module P41, an image quality processing module P42, a landmark detection module P43, an ROI1 determination module P44, a state detection module P45, an ROI2 determination module P46, and a certainty factor visualization module P47. Functions realized by executing the storage module P40, the image reading module P41, the image quality processing module P42, the landmark detection module P43, the ROI1 determination module P44, the state detection module P45, the ROI2 determination module P46, and the certainty factor visualization module P47 are similar to the functions of the storage unit 40, the image reading unit 41, the image quality processing module 42, the landmark detection unit 43, the ROI1 determination unit 44, the state detection unit 45, the ROI2 determination unit 46, and the certainty factor visualization unit 47 of the determination device 4 described above, respectively. The judgment program P4 is a program for causing the judgment device 4 (one or more CPUs) to function as a storage unit 40, an image reading unit 41, an image quality processing unit 42, a landmark detection unit 43, an ROI1 determination unit 44, a state detection unit 45, an ROI2 determination unit 46, and a certainty visualization unit 47.

[0154] The determination device 4 may be configured such that a part or all of it is transmitted via a transmission medium such as a communication line, and is received and stored (including installed) by another device. Furthermore, each module of the determination device 4 may be installed on one of multiple computers, rather than on a single computer. In this case, the series of processes of the determination device 4 described above are performed by a computer system consisting of the multiple computers.

[0155] Next, the function and effect of the determination device 4 will be described.

[0156] The determination device 4 (image processing system 1) includes a landmark detection unit 43 that detects two or more joint landmarks related to a joint in a joint image, which is an in-vivo image of a human joint, and an ROI1 determination unit 44 that determines an ROI1 related to the joint in the joint image based on the two or more detected joint landmarks. In this configuration, the ROI1 is determined based on the two or more detected joint landmarks. This allows for more accurate determination of the ROI1.

[0157] The landmark detection unit 43 may detect two or more joint landmarks in the joint image when the joint image is input using a landmark detection model. In such a configuration, by using the landmark detection model, two or more joint landmarks can be detected more reliably and accurately.

[0158] The landmark detection model may be a model trained based on training data consisting of pairs of the joint image and an image in which two or more of the joint landmarks in the joint image are annotated. In such a configuration, the landmark detection model trained based on the training data can more reliably and accurately detect two or more joint landmarks.

[0159] The landmark detection unit 43 may detect landmarks including the type of each joint landmark, and the ROI1 determination unit 44 may determine the ROI1 further based on the type of each detected joint landmark. In such a configuration, the ROI1 is determined further based on the type of each joint landmark. This allows for a more accurate determination of the ROI1.

[0160] One of the two or more detected joint landmarks may be a bone, and the remaining may be a bone other than the bone, cartilage, fat, joint capsule, or muscle. In such a configuration, a more accurate ROI1 can be determined based on the more specific joint landmark.

[0161] The joint may be a joint of a patient with a blood coagulation disorder or rheumatoid arthritis, preferably a joint of a patient with hemophilia. In this configuration, ROI1 can be determined in an image of a joint of a patient with a blood coagulation disorder or rheumatoid arthritis (preferably a patient with hemophilia).

[0162] The joint image may be an ultrasound image or an X-ray image. Generally, ultrasound examinations or X-ray examinations can be easily performed in outpatient clinics. Therefore, in such a configuration, ROI1 can be easily determined in an outpatient clinic, for example.

[0163] The joint image may be an image with enhanced image quality. In such a configuration, two or more joint landmarks can be detected more accurately based on the image with enhanced image quality, thereby enabling a more accurate determination of the ROI1.

[0164] The joint landmarks may be the tibia, talus, fibula, femur, patella, humerus, radius, ulna, cartilage, fat, joint capsule, or muscle. In such a configuration, a more accurate ROI1 can be determined based on more specific joint landmarks.

[0165] The ROI 1 determination unit 44 may determine the ROI 1 based on the distance from the center of the joint landmark to the boundary line of the ROI 1. In this aspect, the ROI 1 can be determined more reliably.

[0166] The apparatus may further include a condition detection unit 45 that detects the joint condition based on a local joint image, which is an image within the determined ROI 1, among the joint images. In this configuration, the joint condition can be detected more accurately based on the local joint image, thereby allowing the user to more accurately understand the joint condition.

[0167] The state detection unit 45 may detect the joint state of the joint indicated by the local joint image by using a state detection model when the local joint image is input. In such a configuration, the use of the state detection model makes it possible to detect the joint state more reliably and accurately.

[0168] The condition detection model may be a model trained based on training data consisting of pairs of the local joint image and a local joint image associated with a normal label or a local joint image associated with an abnormal location of the joint indicated by the local joint image, In such a configuration, the condition detection model trained based on the training data can detect the joint condition more reliably and accurately.

[0169] The determination device 4 may further include a confidence factor visualization unit 47 that visualizes, in a display mode according to the degree of influence, a portion of the local joint image that has influenced the state detection model when detecting the joint state. In such a configuration, the user can grasp the joint state in more detail from the display mode.

[0170] The determination device 4 may indicate whether the joint condition is normal, abnormal, bleeding, synovitis, or arthropathy. In this manner, a more specific joint condition can be output, allowing the user to understand the more specific joint condition.

[0171] When the joint condition indicates abnormality, bleeding, synovitis, or arthropathy, the determination device 4 may further indicate the corresponding location in the local joint image. In this aspect, the determination device 4 can output the abnormality, bleeding, synovitis, or arthropathy and also output the corresponding location in the local joint image. This allows the user to understand the joint condition more specifically.

[0172] The joint may be an ankle joint, the joint landmark may be a tibia, a fibula, a talus, or fat, and the joint status may indicate normal, abnormal, bleeding, synovitis, or arthropathy. In this aspect, a more accurate ROI1 can be determined for the ankle joint image based on two or more specific joint landmarks. Then, a specific joint status can be output based on the determined ROI1. This allows the user to grasp the more accurate and specific joint status of the ankle joint.

[0173] The joint may be a knee joint, the joint landmark may be a femur, a tibia, a fibula, a patella, or fat, and the joint status may indicate normal, abnormal, bleeding, synovitis, or arthropathy. In this aspect, a more accurate ROI1 can be determined for the knee joint image based on two or more specific joint landmarks. Then, a specific joint status can be output based on the determined ROI1. This allows the user to grasp the more accurate and specific joint status of the knee joint.

[0174] The joint may be an elbow joint, the joint landmark may be a humerus, a radius, an ulna, or fat, and the joint status may indicate normal, abnormal, bleeding, synovitis, or arthropathy. In this aspect, a more accurate ROI1 may be determined for the elbow joint image based on two or more specific joint landmarks. Then, a specific joint status may be output based on the determined ROI1. This allows the user to grasp a more accurate and specific joint status of the elbow joint.

[0175] The image processing system 1 or the determination device 4 may support the diagnosis of the joint condition based on the joint image. In this aspect, it is possible to support the diagnosis of the joint condition based on the joint image.

[0176] [Display Device 5] The display device 5 is a computer device that displays the determination result by the determination device 4. Fig. 26 is a diagram showing an example of the functional configuration of the display device 5. As shown in Fig. 26, the display device 5 is configured to include a storage unit 50 and a result display unit 51.

[0177] Each functional block of the display device 5 is assumed to function within the display device 5, but is not limited to this. For example, some of the functional blocks of the display device 5 may function within a computer device different from the display device 5 and connected to the display device 5 through a network (e.g., the creation device 2, the learning device 3, or the determination device 4), while appropriately transmitting and receiving information with the display device 5. Furthermore, some functional blocks of the display device 5 may be omitted, multiple functional blocks may be integrated into one functional block, or one functional block may be separated into multiple functional blocks.

[0178] FIG. 27 is a diagram showing an example of the hardware configuration of a computer used in the display device 5. As shown in FIG. 27 , the display device 5 is physically configured as a computer system including a CPU 500, a RAM 501, a ROM 502, an input / output device 503 such as a keyboard, a microphone, or a display, a communication module 504 which is a data transmission / reception device, and an auxiliary storage device 505 such as a hard disk or SSD. The CPU 500, the RAM 501, the ROM 502, the input / output device 503, the communication module 504, and the auxiliary storage device 505 may each be configured in plural. The functions of each functional block shown in FIG. 26 are realized by loading predetermined computer software onto hardware such as the CPU 500 and RAM 501 shown in FIG. 27 , which operates the input / output device 503 and the communication module 504 under the control of the CPU 500 and reads and writes data from and to the RAM 501 and the auxiliary storage device 505.

[0179] Hereinafter, each function of the display device 5 shown in FIG. 26 will be described.

[0180] The storage unit 50 stores any information that is used or output in processing of the display device 5. The storage unit 50 may store information calculated by each function of the display device 5. The information stored by the storage unit 50 may be referenced as appropriate by each function of the display device 5, or may be referenced as appropriate via a network by each function of the creation device 2, the learning device 3, or the determination device 4.

[0181] The result display unit 51 visualizes and displays, in a display mode corresponding to the degree of influence, a portion of the local joint image that influenced the state detection model when detecting the joint state, in accordance with an instruction from the certainty visualization unit 47 of the determination device 4. The result display unit 51 may display joint landmarks on the to-be-determined joint image detected by the landmark detection unit 43 of the determination device 4, may display ROI1 on the to-be-determined joint image determined by the ROI1 determination unit 44 of the determination device 4, may display information related to the joint state on the to-be-determined joint image detected by the state detection unit 45 of the determination device 4 (e.g., joint abnormality determination result), or may display ROI2 on the local joint image determined by the ROI2 determination unit 46 of the determination device 4. The result display unit 51 may display output results of each function of the creation device 2, the learning device 3, and the determination device 4 at any timing.

[0182] Next, we will explain the display program P5 that causes a computer to execute a series of processes by the display device 5. The display program P5 is stored in a program storage area formed in the auxiliary storage device 505 provided in the display device 5, for example, as shown in Fig. 28 .

[0183] The display program P5 is configured to include a storage module P50 and a result display module P51. The functions realized by executing the storage module P50 and the result display module P51 are similar to the functions of the storage unit 50 and the result display unit 51 of the display device 5 described above. The display program P5 is a program for causing the display device 5 (one or more CPUs) to function as the storage unit 50 and the result display unit 51.

[0184] The display device 5 may be configured so that part or all of it is transmitted via a transmission medium such as a communication line, and is received and stored (including installed) by another device. Furthermore, each module of the display device 5 may be installed on one of multiple computers, rather than on a single computer. In this case, the series of processes of the display device 5 described above are performed by a computer system consisting of the multiple computers.

[0185] Background: Hemophilia is a congenital bleeding disorder, with intra-articular bleeding being the main problem. Inter-articular bleeding occurs most frequently in the knee, elbow, and ankle joints. Repeated intra-articular bleeding can lead to hemophilic arthropathy, resulting in walking difficulties. Decisions regarding hemostatic treatment for hemophilia are primarily based on the patient's subjective assessment (joint pain, discomfort, etc.), and evaluation of joint condition has not been widely adopted. Furthermore, although joint ultrasound can be easily used in outpatient settings, it has not been widely adopted due to the difficulty of imaging diagnostics for hemophilic arthropathy.

[0186] The image processing system 1 can construct an AI algorithm that uses joint ultrasound images to estimate the presence or absence of joint bleeding and synovitis, and evaluate its diagnostic accuracy. The image processing system 1 provides diagnostic support for joint ultrasound images using AI, allowing appropriate therapeutic intervention to be performed based on an objective assessment of joint bleeding / non-bleeding using joint ultrasound images, rather than a subjective assessment by the patient, which is expected to help hemophilia patients lead healthy and active lives. The establishment of a simple diagnostic method using joint ultrasound examinations using the image processing system 1 is also expected to be widespread among non-specialists.

[0187] The image processing system 1 can use AI to identify synovitis in the ankle, knee, and elbow joints of patients with congenital hemophilia A. The image processing system 1 includes a confidence visualization unit 47, which can display the basis and influence of the AI ​​model's judgment, enabling more accurate information to be provided to physicians. It is generally known that some AI model algorithms (especially deep learning) have poor interpretability, and while the classification results (e.g., abnormal, normal) are correct, they often use incorrect feature values ​​to classify the results. The image processing system 1 allows physicians to reevaluate the AI ​​model's judgment results (landmarks, abnormalities, and their visualization basis) from a clinical perspective, resulting in more accurate clinical decisions. The image processing system 1 is characterized by its ability to annotate and learn landmarks and various types of landmark groups. After detecting landmarks and landmark groups of various types, image processing system 1 extracts ROIs (regions of interest) related to joints, and inputs only localized images of the ROIs (regions of interest) into the next processing module. This has the advantage of higher accuracy and shorter processing time than directly determining abnormalities in the entire image. Image processing system 1 annotates bone surfaces during the learning phase, making it easier for doctors to identify normal joints and to identify bleeding and synovitis.

[0188] 1...image processing system, 2...creation device, 3...learning device, 4...determination device, 5...display device, 20...storage unit, 21...image reading unit, 22...landmark input unit, 23...joint state input unit, 24...input correction unit, 30...storage unit, 31...image reading unit, 32...image quality processing unit, 33...landmark detection model learning unit, 34...ROI1 determination unit, 35...state detection model learning unit, 40...storage unit, 41...image reading unit, 42...image quality processing unit, 43...Landmark detection unit, 44...ROI1 determination unit, 45...State detection unit, 46...ROI2 determination unit, 47...Certainty factor visualization unit, 200, 300, 400, 500...CPU, 201, 301, 401, 501...RAM, 202, 302, 402, 502...ROM, 203, 303, 403, 503...I / O device, 204, 304, 404, 504...Communication module, 205, 305, 405, 505...Auxiliary storage device P2...creation program, P3...learning program, P4...judgment program, P5...display program, P20...storage module, P21...image reading module, P22...landmark input module, P23...joint state input module, P24...input correction module, P30...storage module, P31...image reading module, P32...image quality processing module, P33...landmark detection model learning module, P34...ROI1 determination module, P35...state detection model learning module, P40...storage module, P41...image reading module, P42...image quality processing module, P43...landmark detection module, P44...ROI1 determination module, P45...state detection module, P46...ROI2 determination module, P47...certainty visualization module, P50...storage module, P51...result display module.

Claims

1. An image processing system comprising: a landmark detection unit that detects two or more joint landmarks that are landmarks related to a joint in a joint image, which is an internal image of a human joint; and a determination unit that determines a region of interest related to the joint in the joint image based on the two or more detected joint landmarks.

2. The image processing system according to claim 1, wherein the landmark detection unit detects the joint landmarks using a landmark detection model that detects two or more joint landmarks in an input joint image.

3. The image processing system of claim 2, wherein the landmark detection model is a model trained based on training data consisting of a pair of the joint image and an image in which two or more of the joint landmarks in the joint image are annotated.

4. The image processing system according to any one of claims 1 to 3, wherein the landmark detection unit detects each of the joint landmarks including their types, and the determination unit makes a decision further based on the types of each of the detected joint landmarks.

5. The image processing system according to any one of claims 1 to 4, wherein one of the two or more detected joint landmarks is a bone, and the rest are a bone other than the bone, cartilage, fat, joint capsule or muscle.

6. The image processing system according to any one of claims 1 to 5, wherein the joint is a joint of a hemophilia patient.

7. The image processing system according to any one of claims 1 to 6, wherein the joint image is an ultrasound image or an X-ray image.

8. The image processing system according to any one of claims 1 to 7, wherein the joint image is an image with enhanced image quality.

9. The image processing system according to any one of claims 1 to 8, wherein the joint landmark is a tibia, a talus, a fibula, a femur, a patella, a humerus, a radius, an ulna, cartilage, fat, a joint capsule or a muscle.

10. The image processing system according to any one of claims 1 to 9, wherein the determination unit determines the region of interest based on a distance from a center of the joint landmark to a boundary line of the region of interest.

11. The image processing system according to any one of claims 1 to 10, further comprising a condition detection unit that detects a joint condition, which is a condition of the joint, based on a local joint image, which is an image within the determined region of interest among the joint images.

12. The image processing system according to claim 11, wherein the state detection unit, when receiving the local joint image, detects the joint state of the joint indicated by the local joint image using a state detection model.

13. The image processing system according to claim 12, wherein the condition detection model is a model trained based on training data consisting of pairs of the local joint image and a local joint image associated with a normal label or a local joint image associated with an abnormal location of the joint indicated by the local joint image and an abnormal label.

14. The image processing system according to claim 12 or 13, further comprising a visualization unit that visualizes, in a display mode according to the degree of influence, a portion of the local joint image that has influenced the state detection model when detecting the joint state.

15. The image processing system according to any one of claims 11 to 14, wherein the joint condition indicates whether it is normal, abnormal, bleeding, synovitis, or arthropathy.

16. The image processing system according to claim 15, further comprising, when the joint condition indicates abnormality, bleeding, synovitis or arthropathy, indicating a corresponding location in the local joint image.

17. The image processing system according to any one of claims 11 to 16, wherein the joint is an ankle joint, the joint landmark is a tibia, fibula, talus or fat, and the joint status indicates normal, abnormal, bleeding, synovitis or arthropathy.

18. The image processing system according to any one of claims 11 to 16, wherein the joint is a knee joint, the joint landmark is a femur, a tibia, a fibula, a patella or fat, and the joint condition indicates normal, abnormal, bleeding, synovitis or arthropathy.

19. The image processing system according to any one of claims 11 to 16, wherein the joint is an elbow joint, the joint landmark is a humerus, a radius, an ulna or fat, and the joint condition indicates normal, abnormal, bleeding, synovitis or arthropathy.

20. The image processing system according to any one of claims 11 to 19, further comprising: a diagnostic device for detecting a state of the joint based on the joint image.

21. An image processing program for causing a computer to function as: a landmark detection unit that detects two or more joint landmarks that are landmarks related to a joint in a joint image, which is an internal image of a human joint; and a determination unit that determines a region of interest related to the joint in the joint image based on the two or more detected joint landmarks.

22. The image processing program according to claim 21, wherein the landmark detection unit detects the joint landmarks using a landmark detection model that detects two or more joint landmarks in the joint image when the joint image is input.

23. The image processing program according to claim 22, wherein the landmark detection model is a model trained based on training data consisting of a pair of the joint image and an image in which two or more of the joint landmarks in the joint image are annotated.

24. The image processing program according to any one of claims 21 to 23, wherein the landmark detection unit detects each of the joint landmarks including their types, and the determination unit makes a determination further based on the types of each of the detected joint landmarks.

25. The image processing program according to any one of claims 21 to 24, wherein one of the two or more detected joint landmarks is a bone, and the rest are a bone other than the bone, cartilage, fat, joint capsule, or muscle.

26. The image processing program according to any one of claims 21 to 25, wherein the joint is a joint of a hemophilia patient.

27. The image processing program according to any one of claims 21 to 26, wherein the joint image is an ultrasound image or an X-ray image.

28. The image processing program according to any one of claims 21 to 27, wherein the joint image is an image with enhanced image quality.

29. The image processing program according to any one of claims 21 to 28, wherein the joint landmark is a tibia, a talus, a fibula, a femur, a patella, a humerus, a radius, an ulna, cartilage, fat, a joint capsule or a muscle.

30. The image processing program according to any one of claims 21 to 29, wherein the determination unit determines the region of interest based on a distance from a center of the joint landmark to a boundary line of the region of interest.

31. The image processing program of any one of claims 21 to 30, for causing the computer to further function as a condition detection unit that detects a joint condition, which is the condition of the joint, based on a local joint image, which is an image within the determined region of interest among the joint images.

32. The image processing program according to claim 31, wherein the state detection section, when receiving the local joint image, detects the joint state of the joint indicated by the local joint image using a state detection model.

33. The image processing program according to claim 32, wherein the condition detection model is a model trained based on training data consisting of pairs of the local joint image and a local joint image associated with a normal label or a local joint image associated with an abnormal location of the joint indicated by the local joint image and an abnormal label.

34. The image processing program according to claim 32 or 33, for causing the computer to further function as a visualization unit that visualizes, in a display mode according to the degree of influence, a portion of the local joint image that has been influenced when the condition detection model detects the joint condition.

35. The image processing program according to any one of claims 31 to 34, wherein the joint condition is indicated as normal, abnormal, bleeding, synovitis, or arthropathy.

36. The image processing program according to claim 35, further comprising, when the joint condition indicates abnormality, bleeding, synovitis or arthropathy, indicating a corresponding location in the local joint image.

37. The image processing program according to any one of claims 31 to 36, wherein the joint is an ankle joint, the joint landmark is a tibia, fibula, talus or fat, and the joint condition indicates normal, abnormal, bleeding, synovitis or arthropathy.

38. The image processing program according to any one of claims 31 to 36, wherein the joint is a knee joint, the joint landmark is a femur, a tibia, a fibula, a patella or fat, and the joint condition indicates normal, abnormal, bleeding, synovitis or arthropathy.

39. The image processing program according to any one of claims 31 to 36, wherein the joint is an elbow joint, the joint landmark is a humerus, a radius, an ulna or fat, and the joint condition indicates normal, abnormal, bleeding, synovitis or arthropathy.

40. The image processing program according to any one of claims 31 to 39, which assists in diagnosis of the joint condition based on the joint image.

41. An image processing method executed by a computer, comprising: a landmark detection step of detecting two or more joint landmarks that are landmarks related to a joint in a joint image, which is an internal image of a human joint; and a determination step of determining a region of interest related to the joint in the joint image based on the two or more detected joint landmarks.

42. An image processing device comprising: a landmark detection unit that detects two or more joint landmarks that are landmarks related to a joint in a joint image, which is an internal image of a human joint; and a determination unit that determines a region of interest related to the joint in the joint image based on the two or more detected joint landmarks.

43. The image processing device according to claim 42, further comprising a state detection unit that detects a joint state, which is a state of the joint, based on a local joint image that is an image within the determined region of interest among the joint images.

Images