Generation method, learning method, estimation method, generation device, estimation device, control program, and recording medium

Abstract

Provided is an optimal image conversion method that makes it possible to improve learning accuracy and / or estimation accuracy of a neural network. This image generation device comprises: an acquisition unit that acquires a first image in which at least a part of a first subject appears; an image correction unit and an image conversion unit that perform conversion involving enlargement or reduction of the first image to generate at least one second image; and an output unit that outputs the second image to a learning device as a learning image for use in learning of a machine learning model.

Description

Generation method, learning method, estimation method, generation device, estimation device, control program, and recording medium

[0001] The present disclosure relates to a generation method, a learning method, an estimation method, a generation device, an estimation device, a control program, and a recording medium.

[0002] A technique is known in which an image is input to a neural network (NN) to learn and / or estimate the image.

[0003] Due to limitations in the network configuration and / or VRAM (Video Random Access Memory) capacity, it may be difficult to input an image directly (i.e., at its original size) into a neural network. AlexNet, a representative example of a convolutional neural network (CNN) targeting ImageNet, describes a configuration in which the image is resized so that the short side of the image is 256 pixels long, and then the remaining area is cropped to 256 pixels x 256 pixels (see Non-Patent Document 1). ResNet, another representative example of a CNN, describes a configuration in which the image is randomly resized to a size between 256 and 480 pixels, and then the remaining area is cropped to 224 pixels x 224 pixels (see Non-Patent Document 2).

[0004] Alex Krizhevsky, et al., “ImageNet Classification with Deep Convolutional Neural Networks”, Proc. Advances in Neural Information Processing Systems 25 (NIPS 2012)Kaiming He, et al., “Deep Residual Learning for Image Recognition”, arXiv:1512.03385v1 [cs.CV], December 10, 2015 (https: / / arxiv.org / pdf / 1512.03385.pdf)

[0005] A generation method according to one aspect of the present disclosure includes an acquisition step of acquiring a first image that shows at least a portion of a first subject, a generation step of performing a transformation on the first image that involves enlarging or reducing it to generate at least one second image, and an output step of outputting the second image to a learning device as a learning image to be used in training a machine learning model.

[0006] Furthermore, a learning method according to one aspect of the present disclosure is a method for learning a machine learning model that estimates at least a portion of the characteristics of a second subject from a third image that shows at least a portion of the second subject, in which the learning image generated by the above-described generation method is used as an explanatory variable, and characteristic information indicating the characteristics of at least a portion of the first subject corresponding to the learning image is used as a target variable.

[0007] Furthermore, a machine learning model according to one aspect of the present disclosure is a trained machine learning model generated by the above-described learning method.

[0008] In addition, a generating device according to one aspect of the present disclosure is a generating device that generates learning images to be used for training a machine learning model, and includes an acquisition unit that acquires a first image that shows at least a portion of a first subject, a generating unit that performs a transformation on the first image that involves enlarging or reducing it to generate at least one second image, and an output unit that outputs the second image to the training device as a learning image to be used for training the machine learning model.

[0009] The generating device according to each aspect of the present disclosure may be realized by a computer. In this case, the present disclosure also includes a control program for the generating device that causes the computer to operate as each unit (software element) of the generating device, and a computer-readable recording medium on which the control program is recorded. The computer may also include an edge computer, a cloud server, etc. The functions of the generating device may be realized by multiple devices.

[0010] Furthermore, an estimation method according to one aspect of the present disclosure includes an acquisition step of acquiring a first image that shows at least a portion of a first subject, a generation step of performing a transformation involving enlargement or reduction on the first image based on features of the first image to generate at least one second image, and an estimation step of estimating characteristics of at least a portion of the first subject based on the second image.

[0011] Furthermore, an estimation device according to one aspect of the present disclosure includes an acquisition unit that acquires a first image that shows at least a portion of the first subject, a generation unit that performs a transformation on the first image, including enlarging or reducing it, to generate at least one second image, and an estimation unit that is capable of estimating an estimation result related to characteristics of at least a portion of the first subject based on the second image, wherein the generation unit determines conditions for the transformation based on the first image.

[0012] The estimation device according to each aspect of the present disclosure may be realized by a computer. In this case, the scope of the present disclosure also includes a control program for the estimation device that causes the computer to operate as each unit (software element) of the estimation device, thereby realizing the estimation device on the computer, and a computer-readable recording medium on which the control program is recorded. The computer may also include an edge computer, a cloud server, etc. Furthermore, the functions of the estimation device may be configured to be realized by multiple devices.

[0013] 1 is an example of a block diagram showing the configuration of a learning system according to embodiment 1. FIG. 2 is an example of a flowchart showing the flow of a method for generating a second image for machine learning. FIG. 3 is a schematic diagram showing an example of the flow of clustering processing executed by a selection unit. FIG. 4 is a diagram showing an example of image conversion processing by an image correction unit and an image conversion unit. FIG. 5 is an example of a block diagram showing the configuration of a learning system according to embodiment 2. FIG. 6 is a diagram showing an example of image conversion processing by an image cropping unit, an image correction unit, and an image conversion unit. FIG. 7 is an example of a flowchart showing the flow of a method for generating an image for machine learning. FIG. 8 is a histogram showing an example of processing for limiting a range of pixel values ​​by an image correction unit in a learning system according to embodiment 3. FIG. 9 is a histogram showing another example of processing for limiting a range of pixel values ​​by an image correction unit. FIG. 10 is a histogram showing yet another example of processing for limiting a range of pixel values ​​by an image correction unit. FIG. 11 is a histogram showing yet another example of processing for limiting a range of pixel values ​​by an image correction unit. 10 is a histogram showing yet another example of processing for limiting the range of pixel values ​​by an image correction unit. FIG. 11 is an example of a flowchart showing the flow of a method for generating a second image for machine learning in a learning system according to embodiment 4. FIG. 12 is an example of a flowchart showing the flow of a method for generating a second image for machine learning in a learning system according to embodiment 5. FIG. 13 is an example of a flowchart showing the flow of a method for generating a second image for machine learning in a learning system according to embodiment 6. FIG. 14 is an example of a block diagram showing the configuration of a computer that executes instructions of a program that realizes the control block of the generation device.

[0014] It is known that the optimal resizing method capable of improving the learning accuracy and / or estimation accuracy of a neural network differs depending on the type of image input to the neural network and the type of neural network.

[0015] For example, although there is a known technique for inputting medical images into a neural network to learn and / or estimate the state of the tissues shown in the image, the optimal image conversion method for improving the learning accuracy and / or estimation accuracy of the neural network has not yet been determined.

[0016] First Embodiment An image generating device 1 (hereinafter also simply referred to as "generating device 1") according to an embodiment of the present disclosure will be described in detail below with reference to the drawings.

[0017] The generating device 1 is a generating device that generates training images to be used in training a machine learning model that performs predetermined estimation from a target image. In other words, the generating device 1 is a device that generates training images that serve as training data for the machine learning model. In one aspect, the target image is a medical image.

[0018] The medical images used to generate learning images are usually images actually taken in medical settings. In addition to the subject's object, actual medical images may also contain human tissue other than the part of the body where abnormalities are to be determined, or lesions occurring in parts other than the target part. In other words, the above-mentioned objects are examples of subjects contained in each image, such as medical images.

[0019] The target object may be, for example, a target tissue and / or a target organ. The target tissue may be, for example, at least one of epithelial tissue, connective tissue, muscle tissue, and nervous tissue. The target organ may be, for example, at least one of the digestive system, the cardiovascular system, the endocrine system, and the musculoskeletal system. The target object may be, for example, a joint and / or a bone.

[0020] The type of medical image may be at least one of, but not limited to, an X-ray image, an MRI (Magnetic Resonance Imaging) image, a CT (Computed Tomography) image, and an ultrasound image. The medical image may also be, for example, a dental image. The medical image may be, for example, an inspection device image acquired from an inspection device such as an X-ray inspection device, or an inspection device image subjected to noise reduction processing.

[0021] The image quality of such medical images varies depending on the imaging device, imaging method, imaging conditions, etc. When training a machine learning model, the entire image, including unnecessary parts of the actual medical image (original medical image), is used for training, so training is performed including various elements unrelated to the target learning. Therefore, if the original image is used for training as is, there is a risk of a decrease in training accuracy.

[0022] As described above, the generating device 1 is a generating device that generates training images used to train a machine learning model that performs a predetermined estimation from a target image. In one aspect, the generating device 1 generates medical images for machine learning to train a machine learning model that estimates the bone density of a bone from a target image that shows at least a portion of the bone. Examples of major bone densities include the bone densities of the lumbar vertebrae, femur, calcaneus, and radius. The machine learning model reads the training medical image and learns to output a training image that matches a specific bone density annotated and associated with the medical image.

[0023] Here, the target image is an image in which at least a portion of a predetermined object to be estimated is captured as a subject. The target image may be an X-ray image including a plain X-ray image, an MRI image, a CT image, or the like. Furthermore, the subject captured in the target image may be the same as the subject. The captured regions of the target image and the learning images may include, for example, at least one of the head, neck, chest, lower back, temporomandibular joint, spinal intervertebral joint, hip joint, sacroiliac joint, knee joint, ankle joint, foot, toes, shoulder joint, acromioclavicular joint, elbow joint, wrist joint, hand, fingers, and temporomandibular joint. Furthermore, the target image may be a frontal image or a lateral image.

[0024] The X-ray image data may be a simple X-ray image such as a lumbar X-ray image or a chest X-ray image, or an X-ray image taken using dual-energy X-ray absorptiometry (DXA) or microdensitometry (MD). In a DXA device that measures bone density using the DXA method, when measuring bone density of the lumbar vertebrae, X-rays are irradiated from the front of the subject's lumbar vertebrae. In a DXA device that measures bone density of the proximal femur, X-rays are irradiated from the front of the subject's proximal femur. Here, the terms "front of the lumbar vertebrae" and "front of the proximal femur" refer to the direction that correctly faces the imaging site, such as the lumbar vertebrae or the proximal femur, and may be the ventral side or the back side of the subject's body. The proximal femur includes, for example, at least one of the neck, trochanter, shaft, and the entire proximal femur (neck, trochanter, and shaft). In the MD method, X-rays are irradiated onto the hand. The image data does not have to be an X-ray image, but can be any image containing bone information.

[0025] The machine learning model may learn bone density, etc., of a portion of a bone not shown in a training image of a first subject, based on the training image showing at least a portion of the bone, or may estimate bone density, etc., of a portion of a bone not shown in a target image showing at least a portion of the bone (estimation step). Additionally, the machine learning model may be configured to receive an image (third image) of a predetermined portion of the body of a second subject, and learn and estimate bone density, which is a characteristic of the second subject (estimation unit). The machine learning model may also be configured to receive not only each image but also information about the second subject, such as age and gender, and perform learning and estimation. The machine learning model may also be configured to determine whether the second subject has osteoporosis. The machine learning model may determine osteoporosis based on, for example, proprietary standards or already known guidelines.

[0026] For example, the machine learning model may learn bone density of the femur and / or lumbar bones from training images showing chest bones, or may estimate bone density of the femur and / or lumbar bones from target images showing chest bones.

[0027] Furthermore, for example, the machine learning model may learn the bone density of the femur and / or lumbar bones from a training image showing lumbar bones, or may estimate the bone density of the femur and / or lumbar bones from a target image showing lumbar bones.

[0028] Additionally, the machine learning model may be configured to receive input of images of a predetermined part of the subject's body, and to learn and estimate the subject's bone density. Also, the machine learning model may be configured to receive input of not only each image but also information about the subject, such as age and gender, and to learn and estimate.

[0029] Bone mineral density may be a value related to the density of bone. Bone mineral density is defined as bone mineral density per unit area (g / cm ２ ), bone mineral density per unit volume (g / cm ３ The bone mineral density may be expressed by at least one of the following: YAM (%), T-score, and Z-score. YAM (%) is an abbreviation for "Young Adult Mean" and may be referred to as the young adult mean percentage. For example, bone mineral density may be expressed by bone mineral density per unit area (g / cm ２ ) and YAM (%). The bone mineral density may be an index determined by a guideline or a unique index. For example, the bone mineral density may be a value used in the osteoporosis guidelines (including, but not limited to, the values ​​used in the literature (e.g., "Osteoporosis Prevention and Treatment Guidelines 2015 Edition," compiled by the Osteoporosis Prevention and Treatment Guidelines Editorial Committee, Japan Osteoporosis Society et al., July 10, 2015).

[0030] The bone density may be determined using a DXA device, an X-ray device, an ultrasonic bone density measuring device, etc. Alternatively, the bone density may be a value estimated using a device that estimates bone density from a plain X-ray image.

[0031] Furthermore, the description of the generating device 1 in the present disclosure is not limited to a machine learning model that estimates bone density, but can also be applied to a machine learning model that estimates bone abnormalities. The bone abnormality may be, for example, a bone abnormality related to a bone disease. The bone disease may be, for example, osteoporosis, scoliosis, fracture, spinal stenosis, intervertebral disc degeneration, ankylosing spondylitis, spinal cord injury, osteomyelitis, osteophyte, spinal muscular atrophy, etc.

[0032] The target image may be, for example, an image of a subject captured with an endoscope. More specifically, the target image may include an image captured with an endoscope of a region of the subject including at least one of the nasal cavity, esophagus, stomach, duodenum, rectum, large intestine, small intestine, anus, and colon. The target image of these regions may be used in a machine learning model that outputs analysis results that clearly indicate regions of interest including at least one of inflammation, polyps, and cancer. In such a case, the machine learning model may be a model trained based on, for example, first training images including images with regions of interest and first training data indicating the presence of the regions of interest, and second training images including images without the regions of interest and second training data indicating the absence of the regions of interest.

[0033] The first training image may be a target image for machine learning generated by the generating device 1 for training a machine learning model that outputs an analysis result that clearly identifies a region of interest from a target image. The first training data may include information indicating the inflammation level (degree of inflammation) or malignancy level (degree of malignancy) of the region of interest. The analysis result may be, for example, a display that surrounds the region of interest, a display that points to the region of interest, or a display that superimposes a color on the region of interest. Derivation basis information indicating the basis for deriving the analysis result may be output together with such analysis result.

[0034] The target image may be, for example, an image of the subject's eyes, skin, or the like captured with a digital camera. The target image of these areas may be used, for example, in a machine learning model that outputs analysis results that clearly indicate symptoms of interest. For example, if the symptoms of interest are of the eyes, they may include symptoms indicating diseases including at least one of glaucoma, cataracts, age-related macular degeneration, conjunctivitis, hordeolum, retinopathy, and blepharitis. Alternatively, if the symptoms of interest are of the skin, they may include symptoms including skin cancer, hives, atopic dermatitis, herpes, and the like. The analysis results may be displayed by surrounding these symptoms of interest, by pointing to the symptoms of interest, by superimposing a color on the symptoms of interest, or by displaying the name of the disease.

[0035] The machine learning model may be, for example, a model trained based on first training images having images including an attention sign and first teacher data indicating the presence of the attention sign, and second training images including images without the attention sign and second teacher data indicating the absence of the attention sign. The first training images may be target images for machine learning generated by the generation device 1 for training a machine learning model that outputs analysis results that clearly indicate the attention sign from the target images. Derivation basis information indicating the basis for deriving the analysis results may be output together with such analysis results.

[0036] The predetermined estimation is not limited to estimation of bone density, but may also be estimation of subject characteristics, such as component ratios of bone or other substances, and estimation of disease. Furthermore, the machine learning model may predict a time point different from the current estimation result of the subject characteristics estimated from the image data. The different time point may be a future and / or past time point relative to the time point at which the image data was captured. In the following explanation, the generation device 1 will be described using an example in which each process is performed on a medical image, but the present invention is not limited thereto, and images related to biology, engineering, etc. may also be used.

[0037] The description of the machine learning model in this disclosure is not limited to a configuration that makes a predetermined estimation from medical images, etc., but can also be applied to a machine learning model that determines the identity of an object appearing in an input image.

[0038] FIG. 1 is an example block diagram showing the configuration of a learning system 100 including a generation device 1 according to the present embodiment. As shown in FIG. 1, the learning system 100 includes a generation device 1, a display device 50, and a learning device 70. The generation device 1 includes a control unit 10, a storage unit 20, and an input / output interface (input / output IF) 30. The learning device 70 may be a cloud-based device or an on-premise device installed in a medical facility or a company that provides analysis services.

[0039] The control unit 10 includes one or more processors (not shown) and controls the generation device 1. The storage unit 20 is a storage device that stores various data, and may store original medical image data (original data) 21 and converted image data (learning image data) 22 obtained by performing image conversion processing (described later) on the original medical image. Here, the original medical image data is data of images taken in a medical setting as they are. The storage unit 20 may also store various control programs that are read and executed by the processor.

[0040] The input / output IF 30 is an interface for communicating information with the outside. The input / output IF 30 may be, for example, a wired communication interface such as a USB port, or a wireless communication interface such as Bluetooth (registered trademark) or Wi-Fi (registered trademark). Furthermore, a user may be able to input data to the generating device 1 via a mouse, keyboard, or the like connected to the input / output IF 30. Here, the user refers to a medical professional, technician, or the like who uses the generating device 1, and the same applies hereinafter. Furthermore, the generating device 1 may be connected to the Internet via the input / output IF 30 to communicate information.

[0041] The present disclosure also includes a configuration in which the generating device 1 does not have an input / output IF 30 and is integrated with a display device 50 capable of displaying medical images, etc., or a learning device 70.

[0042] The following describes each unit of the control unit 10. The control unit 10 includes an acquisition unit 11, a selection unit 12, an image correction unit 14, an image conversion unit 15, and an output unit 16.

[0043] The acquisition unit 11 acquires original medical images. Examples of the original medical images include X-ray images. The captured regions of the original medical images may include at least one of the head, neck, chest, lower back, hip joints, knee joints, ankle joints, feet, toes, shoulder joints, elbow joints, wrist joints, hands, fingers, and temporomandibular joints. The original medical images may be simple X-ray images including at least a portion of bones such as the sternum, lumbar vertebrae, or hip neck. The acquisition unit 11 may acquire the original medical images from the storage unit 20 or from an external device or storage medium connected to the generation device 1 via the input / output IF 30. The original medical images and the training images generated from the original medical images are annotated with characteristic information indicating the bone density of the bones depicted in the medical images.

[0044] The selection unit 12 selects original medical images containing bones acquired by the acquisition unit 11 using a method for evaluating the quality or commonality of the images. The selection unit 12 may select images that satisfy predetermined criteria. In one aspect, the selection unit 12 basically selects images whose quality is evaluated to be better than a standard, or whose commonality is evaluated to be higher than a standard. The reason for selecting images with high commonality is that images that are clustered into groups with low commonality with a large number of images are different from typical images and are therefore considered unsuitable for learning a machine learning model.

[0045] As an example, the selection unit 12 evaluates the commonality of images by combining t-Distributed Stochastic Neighbor Embedding (t-SNE) and Density-based spatial clustering of applications with noise (DBSCAN). In this case, t-SNE may be performed before or after DBSCAN. However, the method by which the selection unit 12 evaluates the commonality of images is not limited to a specific method, and another method may be performed between t-SNE and DBSCAN.

[0046] FIG. 3 is a schematic diagram illustrating an example of the flow of the clustering process executed by the selection unit 12. The selection unit 12 performs a fast Fourier transform (FFT) on an original medical image 301 to obtain a spectral image 302. Next, the selection unit 12 dimensionally compresses the spectral image 302 using t-SNE, as shown in image 303. Here, dimensionality compression refers to a process of reducing multidimensional data to a lower number of dimensions so as to minimize the loss of features of the original information. Next, the selection unit 12 clusters image 303 using DBSCAN. As a result, the clustering result shown in image 304 is obtained. In image 304, it can be seen that there is the largest cluster 3041, small clusters 3042 and 3043, and data that does not form clusters. Clusters to which many data belong are determined to have high commonality. Therefore, the selection unit 12 selects only the data included in the largest cluster 3041 and does not select other data.

[0047] Furthermore, images not selected by the selection unit 12 do not need to be used in the subsequent process of generating learning images. From another perspective, the acquisition unit 11 may also acquire images that are not actually used to generate learning images. Furthermore, an image that contains at least a portion of the object to be the subject and is actually used to generate learning images is an example of a first image in the present disclosure.

[0048] That is, the acquisition unit 11 acquires a plurality of specimen images including a first image that shows the bones of the first subject and is actually used to generate a learning image, and the selection unit 12 selects the first image from the plurality of specimen images. That is, the acquisition unit 11 and the selection unit 12 correspond to an acquisition unit that acquires the first images.

[0049] In the above-described clustering process, the selection unit 12 may classify the plurality of specimen images into a plurality of groups based on the distribution of pixel values ​​of each of the plurality of specimen images acquired by the acquisition unit 11, and select a part of the groups that includes at least the first image. The plurality of groups may be classified based on, for example, a predetermined criterion.

[0050] Furthermore, in the clustering process, the selection unit 12 may classify the plurality of specimen images into a plurality of groups based on the result of dimensional compression of the result of analyzing the pixel value distribution of each of the plurality of specimen images.

[0051] The image quality evaluation method may use at least one of the following: average luminance, variance luminance, and BRISQUE (Blind / Referenceless Image Spatial Quality Evaluator). For example, in a method using average luminance and variance luminance, the selector 12 determines that an image whose values ​​fall within a range defined by a predetermined threshold is of high quality. In other words, the selector 12 does not select an image whose values ​​fall outside the range and are therefore outliers. BRISQUE is a method for evaluating the quality of an image using features that can classify the magnitude or type of image distortion. Since X-ray images have a certain range of luminance values, X-ray images whose luminance range is greater than a certain value are determined to have poor image quality. In a method using BRISQUE, images with a predetermined score or less are determined to be of high quality. The image selection method used by the selector 12 may be preset, or may be selectable by the user via the input / output IF 30 each time a learning image is generated.

[0052] The image correction unit 14 performs a predetermined image correction process on the first image selected by the selection unit 12. As a result, the first image is corrected to an image more suitable for training a machine learning model. The process performed by the image correction unit 14 is also called preprocessing.

[0053] The image correction process in the image correction unit 14 includes a conversion process involving enlargement or reduction of the first image. The conversion process involving enlargement or reduction of the first image may be a process of converting the first image so as to increase or decrease the amount of data of the first image in bitmap format. Such data amount is represented by the product of the number of pixels and the color depth of the image.

[0054] The conversion process involving enlargement or reduction of the first image includes resizing of the first image. The target of the resizing process may be the entire first image or a portion of the first image. In the case of a portion, the target of the resizing process may be a predetermined region of the first image. The resizing process may be performed only on a portion of the first image, such as the center, top, or bottom. The resizing process may be performed only on a region in which bones are visible.

[0055] For example, the resolution (spatial resolution) of an X-ray image varies depending on the manufacturer and model of the X-ray imaging device that creates the X-ray image. Therefore, the number of horizontal and vertical pixels indicating the size of an X-ray image varies even when the imaging region is the same. Thus, the sizes of the multiple first images selected by the selection unit 12 vary. In response to this, the image correction unit 14 resizes the first images to make them uniform in size. The image correction unit 14 may also reduce the size of the first images to make them uniform in size to the smallest size among the multiple first images. By training the machine learning model using first images of uniform size, the estimation accuracy of the machine learning model can be improved.

[0056] The image correction unit 14 may enlarge the size of the first image to make it the largest size among the sizes of the multiple first images. The image correction unit 14 may reduce or enlarge the size of the first image to a size within a predetermined range. The predetermined range may be, for example, a range of 256 to 4,320 pixels horizontally and vertically. The image correction unit 14 may reduce or enlarge the size of the first image to make the size of the portion showing bones uniform, for example.

[0057] If a machine learning model is trained using an image whose size has been adjusted without maintaining the aspect ratio, the estimation accuracy of the machine learning model may be reduced. Therefore, the image correction unit 14 may reduce the size of the first image while maintaining the aspect ratio. In this case, the possibility of the estimation accuracy of the machine learning model being reduced can be reduced.

[0058] When reducing the size of the first image, the degree of loss of luminance information contained in the original first image may vary depending on the reduction rate. Therefore, when reducing the size of the first image, the image correction unit 14 may perform image interpolation processing depending on the reduction rate. Image interpolation processing refers to image processing that calculates pixel values ​​between adjacent pixels. The image interpolation processing may be, for example, at least one of bilinear interpolation, nearest neighbor interpolation, average pixel method, bicubic interpolation, etc. Furthermore, when reducing the size of the first image, the image correction unit 14 may perform a process of reducing the size in stages.

[0059] The resizing process increases or decreases the number of pixels in the first image, or in other words, corresponds to a conversion process that increases or decreases the number of pixels in the first image.

[0060] The image correction unit 14 may perform a predetermined image correction process other than resizing on at least a part of the first image selected by the selection unit 12. Here, the image correction unit 14 may perform the image correction process on the entire first image, or may perform the image correction process on only a part of the first image, such as the central part, upper part, or lower part.

[0061] The predetermined image correction process other than the resizing process may be at least one of pixel value normalization, black and white inversion, CLAHE (Contrast Limited Adaptive Histogram Equalization), white balance correction using Retinex with adjust process, unsharp masking, Detail Enhance, histogram equalization, edge enhancement, noise removal, sharpening, registration, and image size adjustment.

[0062] Here, pixel value normalization is a process of reducing the range of possible pixel values ​​to a fixed range, such as 0 to 1. CLAHE is a process of equalizing the histogram for each region of an image to adjust the contrast. Retinex (with adjust) processing is a process of correcting the colors of objects contained in an image to make them as clear as they are seen by the human eye. Unsharp masking is a filter process that emphasizes blurred edges in an image. Detail Enhance is a process that emphasizes the details of an image. Histogram equalization is a process of correcting the histogram of pixel values ​​to make it flatter. Registration is a process of correcting the positional misalignment of multiple images. Image resizing is a process of adjusting the size of an image to a size suitable for input to a machine learning model.

[0063] These image correction processes reduce the apparent variation in the first image, making the bone features in the first image clearer and easier to see.

[0064] Furthermore, the pixel value normalization process may include a luminance normalization process for each of the plurality of first images and a Retinex process.

[0065] The image correction described above can remove noise and adjust the balance of brightness, hue, etc. while preserving the characteristics of the original image. Images corrected in this way become images more suitable for training a machine learning model. The type of image correction performed by the image correction unit 14 may be preset, or may be selectable by the user via the input / output IF 30 each time a training image is generated.

[0066] The image conversion unit 15 generates a new second image to be used as a learning image by performing a predetermined image conversion process on at least a portion of the first image that has been subjected to the predetermined image correction by the image correction unit 14. The process performed by the image conversion unit 15 is also called data augmentation processing. The predetermined image conversion process may be, for example, at least one of brightness change, contrast change, gamma conversion, optical distortion, CLAHE, Channel Shuffle, blur processing, sharpening processing, cropping, rotation, inversion, and scaling.

[0067] Here, gamma conversion is a process of applying a function to an image, the function taking a gamma value indicating the relationship between pixel values ​​and brightness as an argument, and Channel Shuffle is a process of rearranging channels such as RGB.

[0068] Furthermore, the predetermined image conversion process may be configured to randomly execute at least one of the processes described above. The brightness or contrast of the image may also be changed before and after the image conversion process.

[0069] Furthermore, the image conversion unit 15 may convert, for example, at least a part of the image or the entire image of the first image that has been subjected to predetermined image correction by the image correction unit 14. Furthermore, the image conversion unit 15 may convert the images by standardizing the strength and / or type of image conversion, or may randomly change the strength and / or type of image conversion for each image.

[0070] 4 is a diagram showing an example of the image correction process by the image correction unit 14 and the image conversion process by the image conversion unit 15. Image 401 in FIG. 4 is an example of a human chest X-ray. Image 402 is an example of an image obtained by performing resizing on image 401. Images 403 to 405 are examples of images obtained by performing predetermined image conversion on image 402. Images 403 and 404 have been subjected to processing that randomly changes the brightness and / or contrast of the images, and blurring processing, at positions and to different degrees between the images. Image 405 is an image obtained by performing rotation processing on image 402.

[0071] The image conversion process described above generates a large number of new images with various different conditions. That is, the image conversion unit 15 generates one or more first images by randomly converting a single first image within a range of predetermined parameters.

[0072] By training the machine learning model using a large number of new training images, the accuracy of the machine learning model can be improved. Furthermore, as described above, by associating the same bone density with multiple randomly converted images and using them to train the machine learning model, the influence of brightness differences, contrast differences, noise differences, and the like in the target image on the estimation of bone density can be reduced. The second image serving as the training image may be an image generated by the image correction unit 14 or an image generated by the image conversion unit 15. Therefore, the image correction unit 14 and the image conversion unit 15 correspond to a generation unit that generates the training images.

[0073] The type of image conversion performed by the image conversion unit 15 may be set in advance, or may be selectable by the user via the input / output IF 30 each time a learning image is generated.

[0074] The image correction process by the image correction unit 14 and the image conversion process by the image conversion unit 15 are each an example of image processing in the present disclosure.

[0075] The output unit 16 outputs the new second image generated by the image conversion unit 15 to the storage unit 20 or to an external device. The external output destination may be the display device 50, the learning device 70, a storage device such as a database, or another image processing device. Data may also be transmitted to an external device via the Internet.

[0076] The learning device 70 is a device for realizing a machine learning model and learning thereof. The learning device 70 includes a storage unit 701 that stores a parameter set that defines the machine learning model 702, and a learning unit 703 that trains the machine learning model 702. The learning unit 703 trains the machine learning model 702 by updating the values ​​of the parameter set stored in the storage unit 701.

[0077] The learning system 100 may further include an estimation device (estimation unit) capable of estimating an estimation result related to at least some characteristics of the first subject based on the second image output from the image generation device 1.

[0078] Next, the flow of the method S1 for generating second images for machine learning according to this embodiment will be described. Fig. 2 is an example of a flowchart showing the flow of the method S1 for generating second images for machine learning, which is executed by the generation device 1.

[0079] In S11, the acquisition unit 11 acquires a plurality of original medical images, which are specimen images, from the storage unit 20 or the like (acquisition step).

[0080] In S12, the selection unit 12 selects a medical image to be used for training the machine learning model as a first image using a method for evaluating the quality or commonality of each medical image acquired by the acquisition unit 11 (acquisition step).

[0081] In S13, the image correction unit 14 performs a resizing process on the multiple first images selected by the selection unit 12 so that the sizes are uniform (generation step). The image correction unit 14 may also perform a predetermined image correction process other than the resizing process on at least some of the first images.

[0082] In S14, the image conversion unit 15 generates a new second image to be used as a learning image by performing a predetermined image conversion process on at least a part of the first image that has been at least resized by the image correction unit 14 (generation step). However, the above description does not mean that a first image that has not been subjected to image conversion process by the image conversion unit 15 cannot be used as a learning image.

[0083] The learning image generated by the image conversion unit 15 is output by the output unit 16 to the memory unit 20, or to the memory unit 701 of the learning device 70 that stores a parameter set that defines the machine learning model 702 that is the learning target, together with characteristic information indicating bone density that was annotated on the original medical image (output step).

[0084] In addition, in the training method of the machine learning model 702, a training image, which is a second image obtained by performing image correction processing and image conversion processing on a first image showing the bones of a subject, may be used as an explanatory variable, and characteristic information indicating the bone density of the bones of the subject corresponding to the training image may be used as a dependent variable. In the training method of the machine learning model 702, a training image, which is a second image obtained by performing image correction processing and image conversion processing on a first image showing the bones of a subject, may be used as an explanatory variable, and a value indicating the hormone concentration, vitamin D concentration, etc. of the subject corresponding to the training image may be used as a dependent variable. A trained machine learning model is generated by the training method of the machine learning model 702.

[0085] The first image may be an image taken from the front of the subject, or may be an image taken from the side.

[0086] 2 , the method S1 for generating second images for machine learning improves the accuracy of estimation by the machine learning model by using first images with uniform data amounts in bitmap format through a conversion process in the image correction process. Specifically, the resizing process in the image correction process improves the accuracy of estimation by the machine learning model by using first images with uniform sizes through a resizing process in the image correction process. In other words, the generation method S1 can generate training images that can improve the accuracy of the machine learning model when estimating bone density of bones captured in a target image using the machine learning model.

[0087] The size of the first image to be resized may be determined as follows: First, the image corrector 14 sets a plurality of sizes to be resized. The plurality of sizes may be, for example, at least some of 256 pixels x 256 pixels, 512 pixels x 512 pixels, 1024 pixels x 1024 pixels, etc.

[0088] Next, the image correction unit 14 selects one of the multiple sizes and resizes the first image to the selected size. The learning unit 703 uses the resized first image in machine learning to calculate the accuracy of estimation by the machine learning model 702. By performing these processes for each of the multiple sizes, the estimation accuracy corresponding to each of the multiple sizes is calculated. Then, the image correction unit 14 determines the size with the best estimation accuracy as the size of the first image to be unified by the resizing process.

[0089] In the embodiment, when the size of the first image to be aligned by the resizing process is determined from multiple sizes, the absolute error improved from 0.147 to 0.132 and the correlation coefficient improved from 0.616 to 0.813 compared to when the size of the first image to be aligned by the resizing process is fixed.

[0090] Furthermore, the image interpolation process used in the resizing process may be determined as follows: The image correction unit 14 selects one of a plurality of sizes and one of a plurality of image interpolation processes, and resizes the first image to the selected size using the selected image interpolation process. The learning unit 703 uses the resized first image for machine learning to calculate the accuracy of estimation by the machine learning model 702.

[0091] By performing these processes for each of the multiple sizes and each of the multiple image interpolation processes, the image correction unit 14 calculates the estimation accuracy corresponding to each combination of the multiple sizes and the multiple image interpolation processes.The image correction unit 14 then determines the size combination with the best estimation accuracy as the size of the first image to be resized, and determines the image interpolation process of the above combination as the image interpolation process to be used in the resizing process.

[0092] [Embodiment 2] A second embodiment of the present disclosure will be described below. For ease of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and redundant description will not be repeated.

[0093] 5 is an example of a block diagram showing the configuration of a learning system 100A including an image generating device 1A (hereinafter simply referred to as "generating device 1A") according to this embodiment. As shown in FIG. 5, generating device 1A includes a control unit 10A that further includes an image cropping unit 13 in addition to the components of generating device 1 shown in FIG.

[0094] The image cropping unit 13 performs segmentation processing to detect an area (partial area) in the first image selected by the selection unit 12 that shows at least a portion of the first subject, and performs image cropping processing to generate a cropped image cropped from the first image based on the detection result. The image cropping processing may be included in preprocessing. The image correction unit 14 generates a resized image by changing the image size from the cropped image generated by the image cropping unit 13. Furthermore, the image conversion unit 15 performs a predetermined image conversion processing from the resized image generated by the image correction unit 14 to generate a second image that serves as a learning image. The generated second image contains less information other than the first subject than the first image. This makes it possible to generate learning images that can further improve the accuracy of the machine learning model.

[0095] The cropped image may include a detection region, which is a region detected by the segmentation process, and a non-detection region, which is the other region. The image correction unit 14 and the image conversion unit 15 may perform the same image processing on the detection region and the non-detection region, or may perform different processing. For example, the image correction unit 14 may perform image processing that increases the pixel values ​​of the detection region and decreases the pixel values ​​of the non-detection region. Furthermore, if the cropped image includes multiple types of detection regions, the image correction unit 14 and the image conversion unit 15 may perform the same image processing on the multiple types of detection regions, or may perform different processing on the multiple types of detection regions.

[0096] FIG. 6 is a diagram showing an example of the image cropping process by the image cropping unit 13, the image correction process by the image correction unit 14, and the image conversion process by the image conversion unit 15. Image 601 in FIG. 6 is an example of an X-ray photograph of a person's lumbar spine. Image 602 is an example of an image obtained by cropping a region of the lumbar spine from image 601. Image 603 is an example of an image obtained by resizing image 602. Images 604 to 606 are examples of images obtained by performing a predetermined image conversion process on image 603. Images 604 and 605 have undergone a process of randomly changing the brightness and / or contrast of the images, and a blurring process, which are performed at different positions and to different degrees on each image. Image 606 is an image obtained by performing a rotation process on image 603.

[0097] Next, the flow of the learning image generating method S2 according to this embodiment will be described. Fig. 7 is an example of a flowchart showing the flow of the learning image generating method S2 executed by the generating device 1A.

[0098] In S21, the acquisition unit 11 acquires a plurality of original medical images, which are specimen images, from the storage unit 20 or the like (acquisition step).

[0099] In S22, the selection unit 12 selects a medical image to be used for training the machine learning model as a first image using a method for evaluating the quality or commonality of each medical image acquired by the acquisition unit 11 (acquisition step).

[0100] In S23, the image cropping unit 13 detects an area in the first image selected by the selection unit 12 that shows at least a portion of the first subject, and generates a cropped image cropped from the first image based on the detection result (generation step).

[0101] In S24, the image correction unit 14 performs resizing processing on the plurality of cropped images cropped by the image cropping unit 13 so that the sizes are uniform, thereby generating resized images (generation step). The image correction unit 14 may also perform a predetermined image correction processing other than the resizing processing on at least some of the cropped images.

[0102] In S25, the image conversion unit 15 generates a new second image to be used as a learning image (generation step) by performing a predetermined image conversion process on at least a part of the resized image that has been resized by the image correction unit 14. However, the above description does not mean that a resized image that has not been subjected to image conversion process by the image conversion unit 15 cannot be used as a learning image.

[0103] [Embodiment 3] A third embodiment of the present disclosure will be described below. For ease of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiments, and redundant description will not be repeated.

[0104] In the generation device 1 of this embodiment, the image correction process in the image correction unit 14 of the generation device 1 shown in FIG. 1 includes a conversion process involving enlargement or reduction of the first image, and the conversion process involving enlargement or reduction of the first image includes a color depth conversion process that increases or decreases the color depth of the first image instead of a resizing process of the first image. The target of the color depth conversion process may be the entire first image or a portion of the first image. In the case of a portion, the target of the color depth conversion process may be a predetermined region of the first image. The color depth conversion process may be performed only on a portion of the first image, such as the center, upper, or lower portion. The color depth conversion process may be performed only on a region in which bones are visible.

[0105] For example, the color depth (number of gradations) of an X-ray image varies, for example, between 8 bits, 10 bits, and 12 bits, depending on the manufacturer and model of the X-ray imaging device that creates the X-ray image. Therefore, the color depth of an X-ray image varies even when the imaging region is the same. In this way, the color depths of the multiple first images selected by the selection unit 12 vary. In response to this, the image correction unit 14 performs color depth conversion processing on the first images so that the color depths are consistent.

[0106] When a machine learning model is trained using images with various color depths, the estimation accuracy of the machine learning model may be reduced. Therefore, by training the machine learning model using first images with a consistent color depth, the estimation accuracy of the machine learning model can be improved.

[0107] The image corrector 14 may reduce the color depth of the first image to align it with the minimum color depth of the multiple first images. The image corrector 14 may increase the color depth of the first image to align it with the maximum color depth of the multiple first images. The image corrector 14 may reduce or increase the color depth of the first image to a color depth within a predetermined range. The predetermined range may be, for example, a range from 8 bits to 12 bits. The image corrector 14 may reduce or increase the color depth of the first image to align the color depth of, for example, a portion where bones are visible.

[0108] When reducing the color depth of the first image, the degree of loss of luminance information contained in the original first image may vary depending on the reduction rate. Therefore, when reducing the color depth of the first image, the image correction unit 14 may perform image interpolation processing depending on the reduction rate. Image interpolation processing refers to image processing that calculates pixel values ​​between adjacent pixels. The image interpolation processing may be, for example, at least one of bilinear interpolation, nearest neighbor interpolation, average pixel method, bicubic interpolation, etc. Furthermore, when reducing the color depth of the first image, the image correction unit 14 may perform processing to reduce the color depth in stages.

[0109] The image correction unit 14 may perform a predetermined image correction process other than the color depth conversion process on at least a part of the first image selected by the selection unit 12. Here, the image correction unit 14 may perform the image correction process on the entire first image, or may perform the image correction process only on a part of the first image, such as the central part, upper part, or lower part. Details of the predetermined image correction process other than the color depth conversion process are as described above.

[0110] The image correction unit 14 may perform color depth conversion processing after limiting the range of pixel values ​​of the first image. The limiting conditions may be determined based on the first image. The limiting conditions may be determined based on statistics of the pixel values ​​of the first image.

[0111] 8 to 14 are histograms showing an example of the frequency distribution of pixel values ​​in the first image, which is a chest X-ray image with a color depth of 12 bits and a pixel value range of 0 to 4095.

[0112] 8, the median (1673) and maximum (4095) pixel values ​​of the first image are indicated by dashed lines. The number of gradations from the median to the maximum is 4095-1673+1=2423.

[0113] The image correction unit 14 may convert the pixel values ​​of the first image into an 8-bit color depth (0 to 255) after limiting the range of the pixel values ​​to the range from the median to the maximum value shown in Fig. 8. In this case, pixel values ​​equal to or less than the median value are converted to 0, and the maximum pixel value is converted to 255.

[0114] Converting 2423 tones to 256 tones results in a smaller reduction in the number of tones than converting 4096 (12-bit) to 256 (8-bit) tones. Furthermore, the display content of the converted image is similar to that of the original first image. Therefore, by training the machine learning model using a second image that has a small reduction in the number of tones and whose display content is similar to that of the original first image, the estimation accuracy of the machine learning model can be further improved.

[0115] 9, the median (1673) and 98th percentile (3743) of the pixel values ​​of the first image are indicated by dashed lines. The number of gradations from the median to the 98th percentile is 3743-1673+1=2071.

[0116] The maximum pixel value (4095) is often a value (upper outlier) that is outside the range of pixel values ​​representing the subject. If a machine learning model is trained using an image with an upper outlier, it may have a negative impact on the estimation accuracy of the machine learning model.

[0117] Therefore, the image correction unit 14 may convert the pixel values ​​of the first image into an 8-bit color depth (0 to 255) after limiting the range of the pixel values ​​of the first image to the range from the median to the 98th percentile shown in Fig. 9. In this case, pixel values ​​equal to or less than the median are converted to 0, and pixel values ​​equal to or greater than the 98th percentile are converted to 255.

[0118] Converting 2071 tones to 256 tones results in a smaller reduction in the number of tones than converting 4096 (12-bit) to 256 (8-bit) tones. Furthermore, the converted image is an image in which the upper outliers have been corrected. Therefore, by training the machine learning model using the second image in which the reduction in the number of tones is small and the upper outliers have been corrected, the estimation accuracy of the machine learning model can be further improved.

[0119] 10, the 2nd percentile value (139) and the 98th percentile value (3743) of the pixel values ​​of the first image are indicated by dashed lines. The number of gradations from the 2nd percentile value to the 98th percentile value is 3743-139+1=3605.

[0120] The minimum pixel value (0) is often a value (lower outlier) that is outside the range of pixel values ​​representing the subject. If a machine learning model is trained using an image with a lower outlier, it may have a negative impact on the estimation accuracy of the machine learning model.

[0121] Therefore, the image correction unit 14 may convert the pixel values ​​of the first image into an 8-bit color depth (0 to 255) after limiting the range of the pixel values ​​of the first image to the range from the 2nd percentile value to the 98th percentile value shown in Fig. 10. In this case, pixel values ​​equal to or less than the 2nd percentile value are converted to 0, and pixel values ​​equal to or greater than the 98th percentile value are converted to 255.

[0122] Converting 3605 tones to 256 tones results in a smaller reduction in the number of tones than converting 4096 (12-bit) to 256 (8-bit) tones. Furthermore, the converted image is an image in which the lower outliers have been corrected. Therefore, by training the machine learning model using the second image in which the reduction in the number of tones is small and the lower outliers have been corrected, the estimation accuracy of the machine learning model can be further improved.

[0123] 11, the minimum (0) and maximum (4095) pixel values ​​of the first image are indicated by dashed lines. The number of gradations from the minimum to the maximum is 4095-0+1=4096.

[0124] Depending on the image data, a data type with a color depth of 16 bits may contain a first image with a color depth of 12 bits. In this case, the first image contains 2 １２ From 2 １６ There are no pixels with pixel values ​​up to -1.

[0125] Therefore, the image correction unit 14 may convert the pixel values ​​of the first image to an 8-bit color depth (0 to 255) after limiting the range of the pixel values ​​to the range from the minimum value to the maximum value shown in Fig. 11. In this case, pixel values ​​equal to or greater than the maximum value are converted to 255.

[0126] 2 １６ The reduction in the number of tones is significantly less when converting 4096 (12-bit) to 256 than when converting 4096 (16-bit) to 256 (8-bit). Therefore, by training the machine learning model using the second image in which the reduction in the number of tones is significantly less, the estimation accuracy of the machine learning model can be further improved.

[0127] In the example of Fig. 12, the minimum value (0) of the pixel values ​​of the first image and the average value (2047.5) of the minimum and maximum values ​​(4095) are indicated by dashed lines. The number of gradations from the minimum value to the average value is 2047-0+1=2048. In the example of Fig. 13, the average value (2047.5) of the minimum value (0) and maximum value (4095) of the pixel values ​​of the first image and the maximum value are indicated by dashed lines. The number of gradations from the average value to the maximum value is 4095-2048+1=2048.

[0128] 12 and 13, at least one of the upper and lower limits of the range defined by the image correction unit 14 may be a value calculated from a combination of statistics of pixel values ​​of the first image, such as the average of the minimum and maximum values. Examples of statistics include the minimum, maximum, median, percentile, and average values ​​described above, as well as the mode and test statistics.

[0129] In the example of Figure 14, the minimum (2046) and maximum (3764) pixel values ​​in the region of interest of the first image are indicated by dashed lines. The number of gray levels in the region of interest is 3764-2046+1=1719. The region of interest may be set manually by the user or automatically by the segmentation process described above. The region of interest may be, for example, the spine region of the subject.

[0130] The image correction unit 14 may convert the range of pixel values ​​of the first image to an 8-bit color depth (0 to 255) after limiting the range of pixel values ​​of the region of interest shown in Fig. 14. In this case, pixel values ​​of 2046 or less are converted to 0, and pixel values ​​of 3764 or more are converted to 255.

[0131] The reduction in the number of tones is significantly less when converting 1719 to 256 than when converting 4096 (12-bit) to 256 (8-bit). Therefore, by training the machine learning model using the second image in which the reduction in the number of tones is significantly less, the estimation accuracy of the machine learning model can be further improved.

[0132] Furthermore, the shading of the region of interest remains well preserved in the image converted to 256 tones. Therefore, by training the machine learning model using the second image in which the shading of the region of interest remains well preserved, the estimation accuracy of the machine learning model in the region of interest can be further improved.

[0133] Next, the flow of the method S3 for generating second images for machine learning according to this embodiment will be described. Fig. 15 is an example of a flowchart showing the flow of the method S3 for generating second images for machine learning, which is executed by the generation device 1.

[0134] In S31, the acquisition unit 11 acquires a plurality of original medical images, which are specimen images, from the storage unit 20 or the like (acquisition step).

[0135] In S32, the selection unit 12 selects a medical image to be used for training the machine learning model as a first image using a method for evaluating the quality or commonality of each medical image acquired by the acquisition unit 11 (acquisition step).

[0136] In S33, the image correction unit 14 performs color depth conversion processing on the multiple first images selected by the selection unit 12 so that the color depths are uniform (generation step). Furthermore, the image correction unit 14 may perform a predetermined image correction processing other than the color depth conversion processing on at least some of the first images.

[0137] In S34, the image conversion unit 15 generates a new second image to be used as a learning image by performing a predetermined image conversion process on at least a part of the first image on which the image correction unit 14 has performed at least the color depth conversion process (generation step). However, the above description does not mean that a first image on which the image conversion unit 15 has not performed the image conversion process cannot be used as a learning image.

[0138] 15 , the method S3 for generating a second image for machine learning uses a first image with a uniform amount of data in bitmap format through a conversion process in the image correction process, thereby improving the accuracy of estimation by the machine learning model. Specifically, the method S3 generates a learning image that can improve the accuracy of the machine learning model when using the machine learning model to estimate the bone density of bones captured in a target image.

[0139] [Embodiment 4] A fourth embodiment of the present disclosure will be described below. For ease of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiments, and redundant description will not be repeated.

[0140] In the generation device 1 of this embodiment, the image correction processing in the image correction unit 14 of the generation device 1 shown in Figure 1 includes a conversion processing involving enlargement or reduction of the first image, and the conversion processing involving enlargement or reduction of the first image includes a color depth conversion processing that increases or decreases the color depth of the first image instead of a resizing processing of the first image.

[0141] Furthermore, the image correction unit 14 performs color depth conversion processing after limiting the range of pixel values ​​of the first image to each of a plurality of ranges set based on the statistics of pixel values, thereby creating each of a plurality of first images that have been subjected to color depth conversion processing. The plurality of ranges set based on the statistics of pixel values ​​may be different ranges from each other.

[0142] For example, the image correction unit 14 may create a first image in which the range of pixel values ​​of the first image is limited to the range from the minimum value to the maximum value shown in Figure 11 and then converted to an 8-bit color depth, a first image in which the range of pixel values ​​of the first image is limited to the range from the minimum value to the average value of the minimum and maximum values ​​shown in Figure 12 and then converted to an 8-bit color depth, and a first image in which the range of pixel values ​​of the first image is limited to the range from the average value to the maximum value shown in Figure 13 and then converted to an 8-bit color depth.

[0143] As a result, a plurality of first images that have been subjected to color depth conversion processing are created from one first image, and a plurality of second images that serve as training images are created. Therefore, a large number of second images are created from the plurality of first images, and the machine learning model is trained using the large number of second images as training images, thereby improving the accuracy of the machine learning model.

[0144] Next, the flow of the method S4 for generating second images for machine learning according to this embodiment will be described. Fig. 16 is an example of a flowchart showing the flow of the method S4 for generating second images for machine learning, which is executed by the generation device 1.

[0145] In S41, the acquisition unit 11 acquires a plurality of original medical images, which are specimen images, from the storage unit 20 or the like (acquisition step).

[0146] In S42, the selection unit 12 selects a medical image to be used for training the machine learning model as a first image using a method for evaluating the quality or commonality of each medical image acquired by the acquisition unit 11 (acquisition step).

[0147] In S43, the image correction unit 14 limits the range of pixel values ​​of the selected first image to each of a plurality of ranges set based on pixel value statistics. In S44, the image correction unit 14 performs color depth conversion processing on the plurality of first images with limited pixel value ranges so that the color depths are uniform (generation step). Furthermore, the image correction unit 14 may perform a predetermined image correction processing other than the color depth conversion processing on at least a portion of the first image.

[0148] In S45, the image conversion unit 15 generates a new second image to be used as a learning image by performing a predetermined image conversion process on at least a part of the first image on which the image correction unit 14 has performed at least the color depth conversion process (generation step). However, the above description does not mean that a first image on which the image conversion unit 15 has not performed the image conversion process cannot be used as a learning image.

[0149] [Embodiment 5] A fifth embodiment of the present disclosure will be described below. For ease of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiments, and redundant description will not be repeated.

[0150] In the generation device 1 of this embodiment, the image correction process in the image correction unit 14 of the generation device 1 shown in FIG. 1 includes a conversion process involving enlargement or reduction of the first image, and the conversion process involving enlargement or reduction of the first image includes a resizing process of the first image and a color depth conversion process that increases or decreases the color depth of the first image. By performing both the resizing process and the color depth conversion process, the estimation accuracy of the machine learning model can be further improved. Either the resizing process or the color depth conversion process may be performed first.

[0151] Next, the flow of the method S5 for generating second images for machine learning according to this embodiment will be described. Fig. 17 is an example of a flowchart showing the flow of the method S5 for generating second images for machine learning, which is executed by the generation device 1.

[0152] In S51, the acquisition unit 11 acquires a plurality of original medical images, which are specimen images, from the storage unit 20 or the like (acquisition step).

[0153] In S52, the selection unit 12 selects a medical image to be used for training the machine learning model as a first image using a method for evaluating the quality or commonality of each medical image acquired by the acquisition unit 11 (acquisition step).

[0154] In S53, the image correction unit 14 performs color depth conversion processing on the multiple first images selected by the selection unit 12 so that the color depths are uniform (generation step). In S54, the image correction unit 14 performs resizing processing on the multiple first images that have undergone color depth conversion processing so that the sizes are uniform (generation step). Either S53 or S54 may be performed first. Furthermore, the image correction unit 14 may perform a predetermined image correction processing other than the color depth conversion processing and the resizing processing on at least some of the first images.

[0155] In S55, the image conversion unit 15 generates a new second image to be used as a learning image by performing a predetermined image conversion process on at least a part of the first image that has been subjected to at least the color depth conversion process and the resizing process by the image correction unit 14 (generation step). However, the above description does not mean that a first image that has not been subjected to the image conversion process by the image conversion unit 15 cannot be used as a learning image.

[0156] Sixth Embodiment A sixth embodiment of the present disclosure will be described below. For ease of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiments, and redundant description will not be repeated.

[0157] In the generation device 1 of this embodiment, the image correction process in the image correction unit 14 of the generation device 1A shown in FIG. 5 includes a conversion process involving enlargement or reduction of the first image, and the conversion process involving enlargement or reduction of the first image includes a resizing process of the first image and a color depth conversion process that increases or decreases the color depth of the first image. By performing both the resizing process and the color depth conversion process, the estimation accuracy of the machine learning model can be further improved. Either the resizing process or the color depth conversion process may be performed first.

[0158] Next, the flow of the method S6 for generating second images for machine learning according to this embodiment will be described. Fig. 18 is an example of a flowchart showing the flow of the method S6 for generating second images for machine learning, which is executed by the generation device 1A.

[0159] In S61, the acquisition unit 11 acquires a plurality of original medical images, which are specimen images, from the storage unit 20 or the like (acquisition step).

[0160] In S62, the selection unit 12 selects a medical image to be used for training the machine learning model as a first image using a method for evaluating the quality or commonality of each medical image acquired by the acquisition unit 11 (acquisition step).

[0161] In S63, the image corrector 14 performs color depth conversion processing on the first images selected by the selector 12 so that the color depths are uniform (generation step). In S64, the image cropper 13 detects an area in the first images that has undergone color depth conversion processing, in which at least a part of the first subject is captured, and generates a plurality of cropped images that are respectively cropped from the first images based on the detection results (generation step).

[0162] In S65, the image correction unit 14 performs resizing processing on the multiple cropped images cropped by the image cropping unit 13 so that the sizes are uniform, thereby generating multiple resized images (generation step). The image correction unit 14 may also perform a predetermined image correction processing other than the color depth conversion processing and the resizing processing on at least some of the cropped images. Any of S63 to S65 may be executed first.

[0163] In S66, the image conversion unit 15 generates a new second image to be used as a learning image (generation step) by performing a predetermined image conversion process on at least a part of the resized image that has been subjected to at least the color depth conversion process and the resizing process by the image correction unit 14. However, the above description does not mean that a resized image that has not been subjected to the image conversion process by the image conversion unit 15 cannot be used as a learning image.

[0164] In the present disclosure, the subject is described as a human, but the subject is not limited to a human. The subject may be, for example, a non-human mammal such as an equine, feline, canine, bovine, or porcine animal, or may be an animal other than a mammal (e.g., a bird, reptile, amphibian, or fish). In other words, the target image in the present disclosure may be an image showing an animal's bones.

[0165] [Example of Implementation by Software] The control blocks of the image generation device 1 (1A) (particularly the acquisition unit 11, selection unit 12, image cropping unit 13, image correction unit 14, image conversion unit 15, and output unit 16) may be implemented by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, or may be implemented by software. In the latter case, each function of the generation device 1 is implemented, for example, by a computer that executes instructions of a program P, which is software.

[0166] An example of such a computer (hereinafter referred to as computer C) is shown in Figure 19. Computer C includes at least one processor C1 and at least one memory C2. Memory C2 stores a program P for causing computer C to operate as generation device 1. In computer C, processor C1 reads and executes program P from memory C2, thereby realizing each function of generation device 1.

[0167] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PU), a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0168] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input devices such as a keyboard and a mouse, and / or output devices such as a display and a printer.

[0169] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communications network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0170] In addition, some or all of the functions of each of the control blocks can be realized by logic circuits. For example, integrated circuits in which logic circuits that function as each of the control blocks are formed are also included in the scope of the present disclosure. In addition, the functions of each of the control blocks can also be realized by, for example, a quantum computer.

[0171] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0172] The invention according to the present disclosure has been described above based on the drawings and examples. However, the invention according to the present disclosure is not limited to the above-described embodiments. In other words, the invention according to the present disclosure can be modified in various ways within the scope of the present disclosure, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the invention according to the present disclosure. In other words, it should be noted that a person skilled in the art can easily make various modifications or corrections based on the present disclosure. It should also be noted that these modifications or corrections are included in the scope of the present disclosure.

[0173] [Summary] The generation method according to aspect 1 of the present disclosure includes an acquisition step of acquiring a first image that shows at least a portion of a first subject, a generation step of performing a transformation involving enlargement or reduction on the first image to generate at least one second image, and an output step of outputting the second image to a learning device as a learning image to be used in training a machine learning model.

[0174] A generation method according to aspect 2 of the present disclosure may be a method in aspect 1 above, in which the conversion is the execution of a color depth conversion process that increases or decreases the color depth.

[0175] A generation method according to aspect 3 of the present disclosure may be a method in which, in aspect 2 above, the color depth conversion process is a process of limiting the range of pixel values ​​for the first image and increasing or decreasing the color depth.

[0176] A generation method according to aspect 4 of the present disclosure may be a method according to aspect 3 above, in which the limitation conditions are determined based on the first image.

[0177] A generation method according to aspect 5 of the present disclosure may be a method in the above-described aspect 4, wherein the limitation condition is determined based on statistics of pixel values ​​of the first image.

[0178] A generation method according to aspect 6 of the present disclosure may be a method in the above-mentioned aspect 4, wherein the condition for limitation is determined based on statistics of pixel values ​​of a partial region of the first image.

[0179] A generation method according to aspect 7 of the present disclosure may be a method in aspect 6 above, in which the partial region is a region detected by segmentation processing from the first image.

[0180] A generation method according to aspect 8 of the present disclosure may be a method in which, in aspects 5 to 7 above, the statistical quantity of pixel values ​​is selected from the median, minimum, maximum, percentile, and mean of pixel values.

[0181] A generating method according to aspect 9 of the present disclosure may be a method in any one of aspects 1 to 8 above, in which the generating step performs a plurality of the transformations on the first image to generate a plurality of second images.

[0182] A generation method according to aspect 10 of the present disclosure may be a method according to any of aspects 1 to 9 above, in which the conversion is the execution of a resizing process.

[0183] A generating method according to aspect 11 of the present disclosure may be a method in aspect 10 above, wherein the resizing process is a process of generating a resized image by changing the image size from the first image.

[0184] A generation method according to aspect 12 of the present disclosure may be a method in which, in the above-mentioned aspect 11, the resizing process includes at least one of a process of changing the image size while maintaining the aspect ratio of the first image, a process of gradually changing the image size, and a process of changing the image size together with an image interpolation process.

[0185] A generation method according to aspect 13 of the present disclosure may be a method in which, in the above-mentioned aspect 10, the generation step includes a process of determining an area in the first image in which at least a portion of the first subject is captured and generating a cropped image by cropping out the area, and the resizing process is a process of generating a resized image by changing the image size from the cropped image.

[0186] A generation method according to aspect 14 of the present disclosure may be a method in which, in the above-mentioned aspect 13, the resizing process includes at least one of a process of changing the image size while maintaining the aspect ratio of the cropped image, a process of gradually changing the image size, and a process of changing the image size together with an image interpolation process.

[0187] A generation method according to aspect 15 of the present disclosure may be a method in aspects 1 to 14 above, in which the machine learning model estimates, from a third image in which at least a portion of the second subject is visible, characteristics of at least a portion of the second subject visible in the third image, and / or characteristics of a portion of the second subject not visible in the third image.

[0188] A generation method according to aspect 16 of the present disclosure may be a method in which, in aspect 15 above, the third image is a medical image showing bones in the chest or lumbar region of the second subject, and the machine learning model estimates bone density of the femur and / or lumbar bones of the second subject from the third image.

[0189] A generation method according to aspect 17 of the present disclosure may be a method in which, in the above-mentioned aspect 16, the bone density of the bone is at least one of bone mineral density per unit area, bone mineral density per unit volume, percent of young adult mean, T-score, and Z-score.

[0190] A generation method according to aspect 18 of the present disclosure may be a method in which, in the above aspects 1 to 17, the generation step further includes image processing including at least one of a process for changing the brightness of the first image, a process for changing the contrast, a sharpening process, and a blurring process.

[0191] A learning method according to aspect 19 of the present disclosure is a method for learning a machine learning model that estimates characteristics of at least a portion of a second subject from a third image in which at least a portion of the second subject is captured, and uses training images generated by the generation method described in aspects 1 to 18 above as explanatory variables, and uses characteristic information indicating the characteristics of at least a portion of the first subject corresponding to the training images as a target variable.

[0192] The trained machine learning model according to aspect 20 of the present disclosure is a machine learning model generated by the learning method described in aspect 19 above.

[0193] A generating device according to aspect 21 of the present disclosure is a device comprising: an acquisition unit that acquires a first image that shows at least a portion of a first subject; a generation unit that performs a transformation on the first image, including enlarging or reducing it, to generate at least one second image; and an output unit that outputs the second image to a learning device as a learning image to be used in training a machine learning model.

[0194] A control program according to aspect 22 of the present disclosure is a control program for causing a computer to function as the generating device described in aspect 21 above, and may be configured to cause a computer to function as the acquisition unit, the generation unit, and the output unit.

[0195] A recording medium according to aspect 23 of the present disclosure may be a computer-readable non-transitory recording medium on which the control program according to aspect 22 above is recorded.

[0196] An estimation method according to aspect 24 of the present disclosure is a method including an acquisition step of acquiring a first image that shows at least a portion of a first subject, a generation step of performing a transformation involving enlargement or reduction on the first image based on features of the first image to generate at least one second image, and an estimation step of estimating characteristics of at least a portion of the first subject based on the second image.

[0197] A control program according to aspect 25 of the present disclosure is a control program for causing a computer to execute the estimation method described in aspect 24 above, and may be configured to cause a computer to execute the acquisition step, the generation step, and the estimation step.

[0198] An estimation device according to aspect 26 of the present disclosure includes an acquisition unit that acquires a first image that shows at least a portion of the first subject, a generation unit that performs a transformation on the first image, including enlarging or reducing it, to generate at least one second image, and an estimation unit that is capable of estimating an estimation result related to characteristics of at least a portion of the first subject based on the second image, wherein the generation unit determines conditions for the transformation based on the first image.

[0199] A control program according to aspect 27 of the present disclosure is a control program for causing a computer to function as the estimation device described in aspect 26 above, and may be configured to cause a computer to function as the acquisition unit, the generation unit, and the estimation unit.

[0200] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure.

[0201] 1, 1A Image generation device (generation device) 10, 10A Control unit 11 Acquisition unit 12 Selection unit 13 Image cropping unit 14 Image correction unit 15 Image conversion unit 16 Output unit 20 Storage unit 30 Input / output interface (input / output IF) 50 Display device 70 Machine learning model 100, 100A Learning system C Computer C1 Processor C2 Memory

Claims

1. A generation method including: an acquisition step of acquiring a first image that shows at least a portion of a first subject; a generation step of performing a transformation involving enlargement or reduction on the first image to generate at least one second image; and an output step of outputting the second image to a learning device as a training image to be used in training a machine learning model.

2. The method of claim 1, wherein the conversion is a color depth conversion process that increases or decreases the color depth.

3. The generation method according to claim 2, wherein the color depth conversion process is a process of limiting the range of pixel values for the first image and increasing or decreasing the color depth.

4. The generation method according to claim 3, wherein the condition of the limitation is determined based on the first image.

5. The generation method according to claim 4, wherein the condition of the limitation is determined based on statistics of pixel values of the first image.

6. The generation method according to claim 4, wherein the condition of the limitation is determined based on statistics of pixel values of a partial region of the first image.

7. The generation method according to claim 6, wherein the partial region is a region detected from the first image by a segmentation process.

8. A method according to any one of claims 5 to 7, wherein the pixel value statistics are selected from the group consisting of median, minimum, maximum, percentile and mean pixel values.

9. A generating method according to any one of claims 1 to 8, wherein the generating step performs a plurality of the transformations on the first image to generate a plurality of second images.

10. A generating method according to any one of claims 1 to 9, wherein the transformation is the execution of a resizing process.

11. The generating method according to claim 10, wherein the resizing process is a process of generating a resized image by changing the image size of the first image.

12. The generation method according to claim 11, wherein the resizing process includes at least one of a process of changing the image size while maintaining the aspect ratio of the first image, a process of changing the image size in stages, and a process of changing the image size together with an image interpolation process.

13. The generating method described in claim 10, wherein the generating step includes a process of determining an area in the first image that shows at least a portion of the first subject and generating a cropped image by cropping out that area, and the resizing process is a process of generating a resized image by changing the image size from the cropped image.

14. The generation method of claim 13, wherein the resizing process includes at least one of a process of changing the image size while maintaining the aspect ratio of the cropped image, a process of changing the image size in stages, and a process of changing the image size together with an image interpolation process.

15. A generation method described in any one of claims 1 to 14, wherein the machine learning model estimates, from a third image in which at least a portion of the second subject is visible, characteristics of at least a portion of the second subject that is visible in the third image, and / or characteristics of a portion of the second subject that is not visible in the third image.

16. The generation method described in claim 15, wherein the third image is a medical image showing bones in the thoracic or lumbar region of the second subject, and the machine learning model estimates bone density of the femur and / or lumbar bones of the second subject from the third image.

17. The method of claim 16, wherein the bone mineral density of the bone is at least one of bone mineral density per unit area, bone mineral density per unit volume, percent of young adult mean, T-score, and Z-score.

18. A generating method according to any one of claims 1 to 17, wherein the generating step further includes image processing including at least one of processing to change the brightness of the first image, processing to change the contrast, sharpening processing, and blurring processing.

19. A method for training a machine learning model that estimates characteristics of at least a portion of a second subject from a third image in which at least a portion of the second subject is captured, the method using training images generated by the generation method described in any one of claims 1 to 18 as explanatory variables, and using characteristic information indicating the characteristics of at least a portion of the first subject corresponding to the training images as a target variable.

20. A trained machine learning model generated by the learning method described in claim 19.

21. A generation device comprising: an acquisition unit that acquires a first image that shows at least a portion of a first subject; a generation unit that performs a transformation on the first image, including enlarging or reducing it, to generate at least one second image; and an output unit that outputs the second image to a learning device as a training image to be used in training a machine learning model.

22. A control program for causing a computer to function as the generating device according to claim 21, the control program causing a computer to function as the acquisition unit, the generation unit, and the output unit.

23. A computer-readable non-transitory recording medium on which the control program according to claim 22 is recorded.

24. An estimation method comprising: an acquisition step of acquiring a first image showing at least a portion of a first object; a generation step of performing a transformation involving enlargement or reduction on the first image based on features of the first image to generate at least one second image; and an estimation step of estimating characteristics of at least a portion of the first object based on the second image.

25. A control program for causing a computer to execute the estimation method according to claim 24, the control program causing the computer to execute the acquisition step, the generation step, and the estimation step.

26. An estimation device comprising: an acquisition unit that acquires a first image that shows at least a part of a first subject; a generation unit that performs a transformation on the first image, including enlarging or reducing it, to generate at least one second image; and an estimation unit that is capable of estimating an estimation result related to a characteristic of at least a part of the first subject based on the second image, wherein the generation unit determines conditions for the transformation based on the first image.

27. A control program for causing a computer to function as the estimation device according to claim 26, the control program causing a computer to function as the acquisition unit, the generation unit, and the estimation unit.

Images