Information processing device, radiographic system, information processing method, and program

Abstract

To provide an information processing device capable of acquiring subject information from an optical image using a learned model for estimating desired skeleton information, which is obtained by using learning data in the reduced number of pieces as compared with prior arts.SOLUTION: An information processing device includes: an acquisition unit for acquiring an optical image obtained by imaging a subject in an imaging site for a radiographic image: an estimation unit for estimating skeleton information on the subject in the optical image by using the optical image as input data for a second learned model acquired by performing additional learning using skeleton information on a skeleton different from the skeleton indicated by the skeleton information learned by a first learned model for the first learned model for estimating skeleton information on the skeleton of the subject; and a determination unit for determining the subject information including the information on at least one of the laterality and the region of the subject in the optical image, using information on the subject in the optical image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a radiation imaging system, an information processing method, and a program. [Background technology]

[0002] In recent years, in radiography for medical examinations, radiography support using optical images has been implemented, in which the radiography site is optically photographed to obtain an optical image, and additional information obtained by analyzing the optical image is provided to the operator together with a live image. For example, Patent Document 1 proposes a technology that determines the radiography position of the subject from an optical image and outputs information regarding the suitability of the radiography position, thereby enabling efficient radiography to be performed regardless of the skill or experience of the technician. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-199163 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 determines the photographing position of a subject from an optical image. However, the image analysis technology applied to such photographing position determination faces the challenge of further and continuous improvement of accuracy for a wider range of subjects and photographing conditions.

[0005] In general, it is known that in image processing technology, machine learning algorithms using large amounts of training data can create trained models that enable highly accurate recognition and classification. However, it is difficult to collect large amounts of training data, including optical images that contain personal information about patients, from radiography sites for medical examinations.

[0006] Therefore, one object of one embodiment of the present disclosure is to provide an information processing device that can acquire object information from optical images using a trained model that estimates desired skeletal information obtained using a reduced amount of training data compared to conventional methods. [Means for solving the problem]

[0007] An information processing device according to one embodiment of the present disclosure includes an acquisition unit that acquires an optical image obtained by photographing a subject at a radiation image shooting site, an estimation unit that estimates skeletal information of the subject in the optical image by using the optical image as input data for a second trained model obtained by performing additional learning on a first trained model that estimates skeletal information regarding the subject's skeleton using skeletal information regarding a skeleton different from the skeleton indicated by the skeletal information learned by the first trained model, and a determination unit that determines subject information including at least one of information regarding the laterality and location of the subject in the optical image using the skeletal information of the subject in the optical image. Effect of the Invention

[0008] According to one embodiment of the present disclosure, it is possible to obtain object information from optical images using a trained model that estimates desired skeletal information, which is obtained using a reduced amount of training data compared to conventional methods. [Brief description of the drawings]

[0009] [Figure 1] 1 shows a schematic configuration of a radiation imaging system according to an embodiment. [Diagram 2] 1 shows a schematic configuration of an information processing apparatus according to an embodiment. [Diagram 3] 1 is a flowchart showing a processing procedure according to the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining an example of an optical image according to the first embodiment. [Diagram 5] 10A to 10C are diagrams illustrating an example of an output from a skeleton estimation unit according to the first embodiment. [Figure 6]11 is a flowchart showing an example of a subject information determination process according to the first embodiment. [Figure 7] 3 is a diagram for explaining the arrangement of a radiation generating device in a camera coordinate system. FIG. [Figure 8] 4 is a flowchart of an example of processing using object information according to the first embodiment. [Figure 9] 13 is a flowchart showing a processing procedure according to the second embodiment. [Figure 10] 11A to 11C are diagrams illustrating examples of optical images according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of an output from a skeleton estimation unit according to the second embodiment. [Figure 12] 13 is a flowchart showing an example of a subject information determination process according to the second embodiment. [Figure 13] 13A to 13C are diagrams illustrating other examples of optical images according to the second embodiment. [Figure 14] 13 is a flowchart showing another example of the object information determination process according to the second embodiment. [Figure 15] 13 is a flowchart showing a processing procedure according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, exemplary embodiments and examples for carrying out the present disclosure will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments and examples are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. In addition, in the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar.

[0011] In the following, the term radiation can include, for example, electromagnetic radiation such as X-rays and gamma rays, as well as particulate radiation such as alpha rays, beta rays, particle beams, proton beams, heavy ion beams, and meson beams.

[0012] Moreover, the machine learning model refers to a learning model based on a machine learning algorithm. Specific examples of machine learning algorithms include nearest neighbor methods, naive Bayes methods, decision trees, and support vector machines. Also, deep learning, which uses a neural network to generate features and connection weighting coefficients for learning, can be mentioned. Also, as an algorithm using a decision tree, a method using gradient boosting such as LightGBM and XGBoost can be mentioned. Appropriately, any of the above algorithms that can be used can be used in the following embodiments and examples. Also, teacher data refers to learning data, and is composed of a pair of input data and output data. Also, output data of learning data is also called correct answer data.

[0013] Furthermore, a trained model refers to a model that has been trained (learned) in advance using appropriate teacher data (learning data) for a machine learning model that follows any machine learning algorithm such as deep learning. However, although a trained model is obtained in advance using appropriate learning data, this does not mean that no further learning is performed, and additional learning can be performed. Additional learning can also be performed after the device is installed at the site of use.

[0014] (Embodiment) (Schematic configuration) First, a radiation imaging system, an information processing device, and an information processing method according to an embodiment of the present disclosure will be described with reference to Figs. 1 and 2. An embodiment of the present disclosure is applied to a radiation imaging system 100 and an information processing device 200 as shown in Figs. 1 and 2, for example. Fig. 1 shows a schematic configuration of the radiation imaging system 100 according to an embodiment of the present disclosure, and Fig. 2 shows a schematic configuration of the information processing device 200 according to an embodiment of the present disclosure. Note that Fig. 1 shows a state in which the subject O is in a supine position, but the subject O may be in, for example, an upright position or a sitting position. In addition, an imaging table used to support the subject O may be a table according to the position of the subject O.

[0015] The radiation imaging system 100 is provided with an information processing device 200, a radiation generating device 120, a radiation detector 130, and a camera 140. The information processing device 200 is connected to the radiation generating device 120, the radiation detector 130, and the camera 140 and can control them. The information processing device 200 can perform image processing and analysis processing of various images obtained using the radiation detector 130 and the camera 140. The information processing device 200 is connected to an external storage device 160 such as a server via an arbitrary network 150 such as the Internet or an intranet, and can exchange data with the external storage device 160. The external storage device 160 may be directly connected to the information processing device 200.

[0016] The radiation generating device 120 includes, for example, a radiation generator such as a radiation tube, a collimator, a collimator lamp, and the like, and can irradiate a radiation beam under the control of the information processing device 200. The radiation beam irradiated from the radiation generating device 120 passes through the subject O while being attenuated, and enters the radiation detector 130.

[0017] The radiation detector 130 can detect an incident radiation beam and transmit a signal corresponding to the detected radiation beam to the information processing device 200. The radiation detector 130 may be any radiation detector that detects radiation and outputs a corresponding signal, and may be configured using, for example, an FPD (Flat Panel Detector) or the like. The radiation detector 130 may be an indirect conversion type detector that converts radiation into visible light using a scintillator or the like and then converts the visible light into an electrical signal using an optical sensor or the like, or may be a direct conversion type detector that directly converts incident radiation into an electrical signal.

[0018] The camera 140 is an example of an optical device that captures an optical image of the subject O under the control of the information processing device 200, and obtains the optical image. The camera 140 transmits the optical image obtained by capturing the image to the information processing device 200. The camera 140 may have any known configuration, and may be configured as a camera capable of capturing moving images, such as a video camera, or may be configured as a camera that only captures still images. The camera 140 may be configured to capture images using visible light, or may be configured to capture images using invisible light other than radiation, such as infrared light.

[0019] The information processing device 200 is provided with an optical image acquiring unit 201, a skeleton estimation unit 202, a subject information determining unit 203, a consistency determining unit 204, a radiation image acquiring unit 205, an annotation unit 206, and a display control unit 207. The information processing device 200 also is provided with a CPU 231, a storage unit 232, a main memory 233, an operation unit 234, and a display unit 235. The various units of the information processing device 200 are connected via a CPU bus 230 and are capable of exchanging data with one another.

[0020] The optical image acquiring unit 201 can control the camera 140 and acquire an optical image of the subject O captured by the camera 140. The optical image acquiring unit 201 may acquire an optical image of the subject O from the external storage device 160 or an optical device (not shown) connected to the information processing device 200 via an arbitrary network. The optical image acquiring unit 201 may acquire an optical image stored in the storage unit 232.

[0021] The skeleton estimation unit 202 can estimate skeleton information of the human body in the optical image by performing skeleton estimation using the optical image as input data for the trained model. The trained model used by the skeleton estimation unit 202 according to this embodiment can be a trained model generated by additionally training desired data on a general-purpose trained model for estimating skeleton information obtained using a large amount of training data. Here, the desired data may include skeleton information desired in a medical institution, a medical site, or the like where the radiation imaging system 100 is used. Detailed processing by the skeleton estimation unit 202 will be described later.

[0022] The object information determination unit 203 analyzes the estimated skeletal information, and can determine and recognize object information including information indicating a part of the object O that is to be the subject of radiographic imaging in the optical image, information indicating laterality, information indicating direction, etc. Detailed processing by the object information determination unit 203 will be described later.

[0023] The consistency determination unit 204 can determine whether there is consistency between the subject information determined by the subject information determination unit 203 and information on the subject O included in the radiography order. The information processing device 200 can provide the operator with the determination result to assist in determining whether the subject information obtained using the optical image matches the radiography instruction.

[0024] The radiation image acquiring unit 205 controls the radiation generation device 120 and the radiation detector 130 to perform radiography of the subject O and acquire a radiation image of the subject O from the radiation detector 130. The radiation image acquiring unit 205 may also acquire a radiation image of the subject O from the external storage device 160, a radiation detector (not shown) connected to the information processing device 200 via an arbitrary network, or the like. The radiation image acquiring unit 205 may also acquire a radiation image stored in the storage unit 232.

[0025] The annotation unit 206 annotates the object information determined by the object information determination unit 203 onto the radiation image. Here, annotation refers to a process of embedding object information indicating the part, laterality, direction, etc. of the object O into the radiation image. The annotation unit 206 may be configured to be able to annotate the object information onto the optical image.

[0026] The display control unit 207 can control the display of the display unit 235. The display control unit 207 can display, for example, patient information on the patient who is the subject O, imaging conditions, parameters set by the operator, generated optical images and radiation images, determined subject information, analysis information, and the like on the display unit 235. Here, the analysis information may include, for example, segmentation information, and the like. Furthermore, the display control unit 207 can display, in accordance with a desired configuration, any display such as a button or slider for receiving an operation by the operator, a GUI, and the like on the display unit 235.

[0027] The CPU (Central Processing Unit) 231 is an example of a processor that controls the operation of the information processing device 200. The CPU 231 uses the main memory 233 to control the operation of the entire device in accordance with an operation from an operation unit 234 and parameters stored in the storage unit 232. The processor in the information processing device 200 is not limited to the CPU, and may include, for example, a microprocessing unit (MPU) and a graphics processing unit (GPU). The processor may also include a digital signal processor (DSP), a data flow processor (DFP), and a neural processing unit (NPU).

[0028] The storage unit 232 can store various images, data, etc. processed by the information processing device 200. The storage unit 232 can also store patient information, imaging conditions, parameters set by an operator, etc. The storage unit 232 can also store information on a rule-based algorithm for analysis of skeletal information performed by the subject information determination unit 203, etc. The storage unit 232 may be configured with any storage medium, such as an optical disk or a memory. The main memory 233 is configured with a memory, etc., and can be used for temporary data storage, etc.

[0029] In addition, the GPU can perform efficient calculations by processing more data in parallel. Therefore, when learning is performed multiple times using a machine learning algorithm such as deep learning, it is effective to perform processing with the GPU. Therefore, in this embodiment, the information processing device 200 functioning as an example of a learning unit may use a GPU in addition to a CPU for processing. Specifically, when a learning program including a learning model is executed, learning can be performed by the CPU and the GPU working together to perform calculations. In addition, in the processing of the learning unit, calculations may be performed only by the CPU or the GPU. In addition, the estimation process according to this embodiment may also be realized using a GPU like the learning unit. In addition, when the learned model is provided in an external device, the information processing device 200 does not need to function as a learning unit.

[0030] The learning unit may also include an error detection unit and an update unit, both not shown. The error detection unit obtains an error between correct data and output data output from the output layer of the neural network according to input data input to the input layer. The error detection unit may use a loss function to calculate an error between the output data from the neural network and correct data. The update unit updates the connection weighting coefficients between the nodes of the neural network, etc., based on the error obtained by the error detection unit, so as to reduce the error. The update unit updates the connection weighting coefficients, etc., using, for example, an error backpropagation method. The error backpropagation method is a method of adjusting the connection weighting coefficients between the nodes of each neural network so as to reduce the error.

[0031] As the machine learning model according to the present embodiment, for example, FCN (Fully Convolutional Network), SegNet, etc. can be used. As the machine learning model for performing object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, as the machine learning model for performing object recognition in units of regions, YOLO (You Only Look Once), SSD (Single Shot Detector, or Single Shot MultiBox Detector) can be used.

[0032] The operation unit 234 includes input devices for operating the information processing device 200, such as a keyboard and a mouse. The operator can also input parameters and the like related to a rule-based algorithm of the analysis process using skeletal information performed by the subject information determination unit 203 via the operation unit 234.

[0033] The display unit 235 includes, for example, any display, and displays various information such as subject information and various images under the control of the display control unit 207. The display unit 235 may be, for example, a monitor of a console that operates a radiation imaging apparatus including the radiation generation device 120 and the radiation detector 130. The display unit 235 may be a sub-monitor installed at a position where the operator can observe while assisting in positioning the subject O, or a console monitor of a radiation irradiator. The display unit 235 may be a head-mounted display that the operator can wear while working, or the like, which allows the operator to reliably check the display with little eye movement. The display unit 235 may be configured with a touch panel display, and in this case, the display unit 235 can also be used as the operation unit 234.

[0034] The information processing device 200 can be configured by a computer provided with a processor and a memory. The information processing device 200 may be configured by a general computer, or may be configured by a computer dedicated to a radiation imaging system. The information processing device 200 may be, for example, a personal computer (PC), and a desktop PC, a notebook PC, or a tablet PC (portable information terminal), etc. may be used. Furthermore, the information processing device 200 may be configured as a cloud-type computer in which some components are arranged in an external device.

[0035] Moreover, the optical image acquiring unit 201, the skeleton estimating unit 202, the object information determining unit 203, the consistency determining unit 204, the radiation image acquiring unit 205, the annotation unit 206, and the display control unit 207 may be configured as software modules executed by the CPU 231. Furthermore, each of these components may be configured as a circuit performing a specific function, such as an ASIC, or an independent device.

[0036] Next, the operation of the information processing device 200 under the control of the CPU 231 will be described. First, based on the imaging order information transmitted from an information management device (not shown), the information processing device 200 starts preparation for imaging, and the optical image acquisition unit 201 starts acquiring optical images under the control of the CPU 231. Here, the imaging order information is information corresponding to the unit of examination ordered by a doctor, and is information including, for example, patient information, imaging (scheduled) date and time, and the part, direction, and posture of the imaging target based on the doctor's findings. The imaging order information includes, as information necessary for radiation imaging, for example, the type of radiation detection device to be used (for standing, for lying, portable, etc.), the patient's posture (imaging part, direction, etc.), and radiation imaging conditions (tube voltage, tube current, presence or absence of grid, etc.).

[0037] The optical image acquiring unit 201 controls the camera 140 to capture an optical image of the subject O, and acquires the optical image from the camera 140. The optical image acquired by the optical image acquiring unit 201 is transferred sequentially to the main memory 233, the skeleton estimation unit 202, and the subject information determination unit 203 via the CPU bus 230.

[0038] The skeleton estimation unit 202 estimates skeleton information of the subject O using the transferred optical image as input data for the trained model. Then, the subject information determination unit 203 obtains the subject information from the estimated skeleton information. The subject information is transferred to the consistency determination unit 204 via the CPU bus 230. The consistency determination unit 204 compares the imaging order information with the subject information and outputs a consistency determination result. The optical image, skeleton information, subject information, and consistency determination result are transferred to the storage unit 232 and the display control unit 207 via the CPU bus 230. The storage unit 232 stores the transferred various information. The display control unit 207 causes the display unit 235 to display the transferred various information.

[0039] The operator checks the various types of displayed information and issues operation instructions as necessary via the operation unit 234. For example, if the consistency determination result is correct, the operator issues an instruction to capture a radiographic image via the operation unit 234. This instruction to capture a radiographic image is transmitted to the radiographic image acquisition unit 205 by the CPU 231.

[0040] Upon receiving an instruction to perform radiography, the radiographic image acquiring unit 205 controls the radiation generating device 120 and the radiation detector 130 to perform radiography. In radiography, a radiation beam is first irradiated from the radiation generating device 120 toward the subject O, and the radiation beam that passes through the subject O while attenuating is detected by the radiation detector 130. The radiographic image acquiring unit 205 acquires a signal corresponding to the intensity of the radiation beam detected by the radiation detector 130 as a radiographic image. Data of this radiographic image is transferred sequentially to the main memory 233 and the annotation unit 206 via the CPU bus 230.

[0041] The annotation unit 206 annotates the transferred radiographic image with the subject information stored in the storage unit 232. The annotated radiographic image is transferred to the storage unit 232 and the display control unit 207 via the CPU bus 230. The storage unit 232 stores the transferred annotated radiographic image. The display control unit 207 displays the transferred annotated radiographic image on the display unit 235. The operator can check the displayed radiographic image and issue operation instructions via the operation unit 234 as necessary.

[0042] Example 1 Hereinafter, a radiation imaging system, an information processing device, and an information processing method according to a first embodiment of the present disclosure will be described with reference to Fig. 3 to Fig. 7. In this embodiment, a process of recognizing the laterality of a subject to be imaged from an optical image acquired at a predetermined frame rate using a video camera as the camera 140, and outputting the result as subject information will be described. Laterality is information indicating whether a body part to be imaged is on the left or right side, with respect to a body part having left and right sides.

[0043] (Processing flow) A series of processing steps according to this embodiment will be described below with reference to Fig. 3. Fig. 3 is a flowchart showing the processing steps according to this embodiment. When the processing steps according to this embodiment are started, the process proceeds to step S301.

[0044] (Step S301) In step S301, the optical image acquisition unit 201 controls the camera 140 to acquire optical images of the radiation imaging site including the subject of radiation imaging. In this embodiment, the camera 140 is a video camera attached to a radiation generator, and captures an image of the subject taking an imaging posture on the radiation detector 130 arranged on a supine table, and outputs optical images at a predetermined frame rate.

[0045] Here, a case where the radiation detector 130, a right hand 402, and a left hand 403 are captured in the optical image 400 will be described with reference to FIG. 4. FIG. 4 shows an example of an optical image according to this embodiment. Here, a radiation detector region 401 in the optical image 400 is a region showing the radiation detector 130. Furthermore, in the example shown in FIG. 4, light from a collimator lamp is irradiated onto the right hand of the subject, and a collimator lamp irradiation region 404 is depicted on the right hand 402 in the optical image 400. Here, the collimator lamp is a device that irradiates visible light and is attached to a collimator in order to confirm a radiation irradiation target region, which is a region to be irradiated with radiation, before radiation irradiation. The collimator lamp irradiation region 404 generated by the collimator lamp coincides with the radiation irradiation target region during radiation imaging.

[0046] (Step S302) In step S302, the skeleton estimation unit 202 estimates skeleton information of the subject by using the optical image acquired by the optical image acquisition unit 201 as input data of the trained model. Here, the skeleton information is information indicating a plurality of predefined feature points on the subject, and includes coordinates indicating parts such as the face, shoulders, hands, waist, and feet, and can be used to recognize the posture of the human body. In this embodiment, the skeleton estimation unit 202 uses a trained model composed of a machine learning model represented by a neural network. Here, the trained model is a model generated in advance by supervised machine learning using various image data not limited to the site of radiography, and additionally learns data corresponding to an output desired at the site of radiography.

[0047] First, in the head, the feature points can include at least one of the eyes, nose, and mouth, for example, in order to distinguish the front and rear of the head. In addition, in the torso, the feature points can include, for example, feature points for distinguishing the neck, chest (thoracic vertebrae), abdomen, and lumbar (lumbar vertebrae). Furthermore, in the limbs, the feature points can include, for example, shoulders, elbows, wrists, hip joints, knees, and ankles. The skeleton estimation unit 202 estimates skeleton information indicating these feature points, so that the information processing device 200 can recognize a wide range of postures. In addition, parts having laterality can be defined as different feature points at the time of skeleton estimation, such as the right hand and the left hand. In this case, the information processing device 200 can estimate laterality based on the feature points.

[0048] The skeleton estimation process may be performed on the optical image using one neural network model, or may include a subject region extraction process for extracting a subject region from the optical image, and perform skeleton estimation from the extracted subject region. In this case, the subject region in the optical image may be used as input data for the trained model. The subject region in the optical image may be used as the training data. Furthermore, the skeleton estimation process may include a class classification process for classifying whether the optical image shows the subject's whole body or only the head, hands, or feet. In this case, the skeleton estimation unit 202 may select and apply a trained model for skeleton estimation by class, specialized for the whole body, head, arms, or feet, based on the result of the class classification process.

[0049] In general, as a neural network model for skeletal estimation, a trained model for general-purpose use is used, which is machine-learned using a large amount of data including an optical image and skeletal information of the subject in the optical image. However, when such a trained model is used as a skeleton estimation unit required in the medical field of this embodiment, for example, there is a possibility that the feature points that can be output may be insufficient. For example, it is assumed that the shoulder joint, elbow joint, wrist, hip joint, knee joint, and ankle are trained to be output for the limbs, but the feature points representing the fingers, toes, and the front and back of the fingers and toes cannot be output. Therefore, in this embodiment, a trained model that has been additionally trained on such a general-purpose trained model is used so that the feature points required for each medical institution or medical field can be output. The additional training of the neural network model for additionally outputting feature points related to some human body parts can be realized by machine learning using a relatively small dataset compared to the dataset used for training the general-purpose trained model. For example, additional training can be performed with a dataset necessary and sufficient for outputting feature points required for each medical institution or medical field where the radiation imaging system 100 is used.

[0050] The learning data used for the additional learning may include a data set in which an optical image is used as input data, and for a feature point desired for additional learning, data in which a label indicating the feature point is attached to the position of the feature point in the optical image, or an image is used as output data (correct answer data). The correct answer data may be generated by a doctor or the like based on the optical image. When an optical image is input, a trained model trained using such learning data can output the coordinates of each feature point, including the additionally trained feature point, in the input optical image, and a probability indicating the feature point-likeliness (likelihood). If the probability is 0.0, the trained model does not need to output the position of the feature point. In addition, the training data of a general-purpose trained model before additional learning may also include a data set of the same format, but the training data used for additional learning includes skeletal information regarding a skeleton different from the skeleton indicated by the skeletal information trained by the general-purpose trained model. In addition, the trained model used by the skeleton estimation unit 202 may be one that has already undergone additional learning, and it is not necessary to perform learning every time processing is performed.

[0051] In this embodiment, an example will be described in which the skeleton estimation unit 202 applies arm skeleton estimation, which estimates the left wrist, left elbow joint, left shoulder joint, right wrist, right elbow joint, and right shoulder joint as skeleton information, to the optical image 400. In this example, the skeleton estimation unit 202 inputs the optical image 400 to the trained model. In this case, the trained model detects the left wrist 501 and the right wrist 502 as shown in FIG. 5, and outputs their respective coordinates (x501, y501), (x502, y502) and probabilities P501 and P502 representing the likelihood of each being a feature point. In contrast, the trained model estimates that the probability that the left shoulder joint, left elbow joint, and right shoulder joint are included in the optical image 400 is 0.0, and does not output the coordinates corresponding to each. On the other hand, the trained model estimates that the right elbow joint 503 is present at the coordinates (x503, y503) with a probability of P503. Here, it is assumed that the estimation regarding the right elbow joint is incorrect, and in this case, the probability P503 will be smaller than the probabilities P501 and P502.

[0052] The skeleton estimation unit 202 outputs the coordinates and probability of each feature point in a format like table 500 as skeleton information. The skeleton estimation unit 202 may output the skeleton information in a format according to the output of the trained model. For example, the trained model may output matrix data in a format like table 500, and the skeleton estimation unit 202 may output the matrix data or may output it in a format like table 500. Also, for example, the trained model may output a map (heat map) that visualizes the feature amounts extracted from the optical image. In this case, the skeleton estimation unit 202 may output the map, or may output the position and probability of the most probable feature point in the map in a format like table 500.

[0053] (Step S303) In step S303, the subject information determination unit 203 determines the laterality of the subject, who is the subject of the radiographic image, from the skeletal information estimated by the skeleton estimation unit 202, and outputs it as subject information. Specifically, the subject information determination unit 203 applies a rule-based algorithm as shown in Fig. 6 to the skeletal information estimated by the skeleton estimation unit 202. In this rule-based algorithm, the laterality of the subject is determined based on the probability and coordinates of the skeletal information output by the skeleton estimation unit 202.

[0054] Here, an example will be described in which the rule-based algorithm in Fig. 6 is applied to the output of the skeleton estimation shown in Fig. 5. The output of the skeleton estimation shown in Fig. 5 includes a left wrist 501, a right wrist 502, and a right elbow joint 503, and the probability P503 of the right elbow joint 503 is assumed to be small.

[0055] First, in step S601, the object information determining unit 203 deletes feature points whose probability of being a feature point is less than a threshold value from among the feature points included in the skeleton information output by the skeleton estimation unit 202. In the above example, the object information determining unit 203 deletes the right elbow joint 503 as a feature point because the probability P503 of the right elbow joint 503 is less than the threshold value.

[0056] Next, in step S602, the subject information determination unit 203 determines whether the number of remaining feature points is one or more. If it is determined in step S602 that the number of feature points is more than one, the process proceeds to step S603. On the other hand, if it is determined in step S602 that the number of feature points is one, the process proceeds to step S604. In the above example, the subject information determination unit 203 determines that the number of remaining feature points is two, that is, the left wrist 501 and the right wrist 502, and therefore the process proceeds to step S603.

[0057] In step S603, the subject information determination unit 203 obtains the position of a region to be irradiated with radiation (irradiation region) in the optical image, and deletes, from the remaining feature points, feature points other than the feature point closest to the position of the region to be irradiated with radiation. In the above example, the subject information determination unit 203 calculates the distance between the region to be irradiated with radiation and the coordinates (x501, y501) of the left wrist, and the distance between the region to be irradiated with radiation and the coordinates (x502, y502) of the right wrist.

[0058] Here, the position of the region to be irradiated with radiation in the optical image 400 may be the position of the collimator lamp irradiation region 404. Therefore, the object information determination unit 203 may extract the collimator lamp irradiation region 404 by, for example, threshold processing based on the brightness of the optical image, and calculate the distance from the feature point by regarding the collimator lamp irradiation region 404 as the region to be irradiated with radiation. Note that the method of extracting the collimator lamp irradiation region 404 in the optical image is not limited to this, and any known method may be used. For example, the object information determination unit 203 may extract the collimator lamp irradiation region 404 by using edge detection, corner detection, or the like.

[0059] The distance dn between the coordinates (xn, yn) of the characteristic point n and the representative point (for example, the center position (xc, yc)) of the region to be irradiated with radiation can be calculated according to the following formula 1.

number

[0060] The subject information determination unit 203 compares the distance between each feature point and the planned radiation irradiation area, which is calculated using Equation 1, finds the feature point that is closest to the planned radiation irradiation area, and can delete feature points other than the found feature point. In the above example, the right wrist 502 is closest to the planned radiation irradiation area among the outputs of the skeleton estimation shown in Fig. 5, so the subject information determination unit 203 deletes the left wrist 501, which is a feature point.

[0061] In step S604, the subject information determination unit 203 outputs the laterality of the remaining feature points as subject information. In the above example, the subject information determination unit 203 outputs the laterality "right" of the right wrist 502, which is the remaining feature point. When the laterality of the feature points is output in step S604, the laterality determination process, which is the subject information determination process according to this embodiment, ends.

[0062] In the subject information determination process, the correspondence between the remaining feature points and the feature points to be output may be determined in advance by a rule. For example, if the right elbow joint or the right wrist remains as a feature point, the laterality to be output may be set to "right," and if the left elbow joint or the left wrist remains as a feature point, the laterality to be output may be set to "left."

[0063] Although the above describes the case where the feature points are the right and left wrists, other features such as shoulder joints, elbow joints, hip joints, knee joints, or ankles may also be used. In this case, too, the subject information determining unit 203 can output, as subject information, the laterality of the feature points included in the skeleton information output by the skeleton estimation unit 202.

[0064] Furthermore, the rules in the subject information determination process are not limited to the above rules. For example, in the subject information determination process, if the remaining feature point is a "right shoulder joint" or a "left shoulder joint," and the laterality to be determined is a laterality related to a "hand," some kind of error may occur, so a rule may be considered such that an error is output.

[0065] The region to be irradiated with radiation is not limited to the collimator lamp irradiation region, and may be obtained using other known methods. For example, more simply, the center of the optical image may be determined as the region to be irradiated with radiation. In this case, the subject information determining unit 203 may output the laterality of a feature point close to the center of the optical image as the subject information. In this case, the laterality of the subject can be obtained even if the collimator lamp irradiation region is not captured in the optical image. In addition, for example, a marker or the like indicating the region to be irradiated with radiation may be provided on the imaging table, and the subject information determining unit 203 may extract the marker from the optical image to obtain the region to be irradiated with radiation.

[0066] Furthermore, if higher accuracy is required in determining the object information, the planned radiation irradiation area in the image may be calculated taking into account the spatial positions of the radiation generation device 120 and the camera 140. For example, as shown in Fig. 7, consider a camera coordinate system in which the optical center of the camera 140 is set as the origin (0,0,0), the optical axis direction of the camera 140 is set as the Z axis direction, and the horizontal and vertical directions of the image are set as the X axis direction and the Y axis direction, respectively. Here, consider a case in which the spatial coordinates (X120, Y120, Z120) of the radiation generation device 120 and the radiation irradiation direction (vx, vy, vz) are known.

[0067] These pieces of information can be obtained by, for example, installing the camera 140 with respect to the radiation generating device 120 while measuring the position with a tape measure or the like. The information may also be obtained based on the amount of drive from the installation position of the radiation generating device 120 using a configuration capable of mechanically grasping the amount of drive, such as a stepping motor. Furthermore, the information may also be obtained by acquiring the amount of displacement of the position and angle from the initial position by attaching a gyro mechanism, an acceleration sensor, or the like to the radiation generating device 120, and inputting the acquired information to the information processing device 200. However, when the camera 140 is installed with respect to the radiation generating device 120, the spatial coordinates (X120, Y120, Z120) and the radiation irradiation direction (vx, vy, vz) of the radiation generating device 120 in the camera coordinate system after installation may generally be considered to be fixed values. In this case, the coordinates (xp, yp) of the area to be irradiated with radiation in the acquired optical image can be expressed by the following equation 2, where D is the distance between the radiation generating device 120 and the radiation detector 130, (fx, fy) is the focal length of the camera 140, and (cx, cy) is the optical center.

number

[0068] By determining the planned radiation irradiation region (xp, yp) as described above, the subject information determination unit 203 can more accurately determine the laterality of the subject by outputting the laterality of the feature point closest to the planned radiation irradiation region (xp, yp) as subject information.

[0069] (Example of use of determined specimen information) An example of processing using the subject information obtained above will be described with reference to Fig. 8. In this example, when processing using the subject information is started, the subject information obtained in the subject information determination processing is transferred to the consistency determination unit 204 via, for example, the CPU bus 230, and the processing proceeds to step S801. In step S801, the consistency determination unit 204 performs consistency determination between the imaging order information and the subject information. For example, when the imaging target part of the imaging order is "hand" and the laterality is "right", if the laterality included in the obtained subject information is "right", the consistency determination unit 204 outputs "consistency", and otherwise outputs "inconsistency".

[0070] In step S802, the display control unit 207 causes the display unit 235 to display the optical image acquired by the optical image acquisition unit 201, the object information output by the object information determination unit 203, the imaging order, and the consistency determination result. The display control unit 207 can also cause the display unit 235 to display the object region and skeletal information acquired by the skeleton estimation unit 202, and analysis information obtained during the object information determination process, etc.

[0071] In step S803, after the operator checks the information displayed on the display unit 235, the operator inputs a radiographic image capturing instruction to the information processing device 200 via the operation unit 234. The radiographic image acquiring unit 205 acquires a radiographic image in response to the radiographic image capturing instruction from the operator. Specifically, the radiographic image acquiring unit 205 causes the radiation generating device 120 to irradiate a radiation beam under imaging conditions according to the imaging order, and acquires a radiographic image from the radiation detector 130 that detects the radiation transmitted through the subject O. The radiographic image acquiring unit 205 may acquire a radiographic image in response to the completion of the consistency determination process. In this case, the radiographic image acquiring unit 205 may acquire a radiographic image corresponding to the optical image used in the subject information determination process from the external storage device 160, the storage unit 232, etc., for example.

[0072] In step S804, the subject information is transferred to the annotation unit 206 via, for example, the CPU bus 230, and the annotation unit 206 annotates the acquired radiographic image with the subject information. For example, since the radiographic image is a fluoroscopic image, even if the radiographic image is a radiographic image of a right hand, it is difficult to determine whether the hand is right or left from the radiographic image. However, the annotation unit 206 can add information that the subject shown in the radiographic image is a right hand by arranging information such as "right" as annotation information on the radiographic image as laterality information. The radiographic image annotated with the subject information can be transferred to the display control unit 207 or the storage unit 232 via the CPU bus 230. Note that the annotation unit 206 may annotate the subject information on the optical image, and in this case, the optical image annotated with the subject information can be transferred to the display control unit 207 or the storage unit 232 via the CPU bus 230.

[0073] In step S805, the display control unit 207 can display the radiation image annotated with the object information on the display unit 235. The display control unit 207 may display the optical image and the radiation image annotated with the object information side by side or in a switched manner on the display unit 235. When the display process of the radiation image is completed, the process using the object information is completed. Furthermore, the display unit 235 may display the optical image annotated with the object information as the optical image.

[0074] As described above, the radiation imaging system 100 according to this embodiment includes the radiation generating device 120 and the radiation detector 130 for performing radiation imaging of the subject, the camera 140 functioning as an example of an optical device for capturing an optical image of the subject, and the information processing device 200. The information processing device 200 includes an optical image acquiring unit 201, a skeleton estimating unit 202, and a subject information determining unit 203. The optical image acquiring unit 201 functions as an example of an acquiring unit for acquiring an optical image obtained by capturing an image of the subject at a site where the radiation image is captured. The skeleton estimating unit 202 functions as an example of an estimating unit for estimating skeletal information of the subject in the acquired optical image by using the acquired optical image as input data for a second trained model obtained by performing additional learning using skeleton information on a skeleton different from the skeleton indicated by the skeleton information learned by the first trained model for a general-purpose trained model for estimating skeletal information on the skeleton of the subject. The object information determining section 203 functions as an example of a determining section that determines object information including information on the laterality of the object in the optical image, using skeletal information of the object in the optical image.

[0075] With the above configuration, the information processing device 200 according to this embodiment can determine the laterality of the subject (radiography subject) as subject information by using skeletal information estimated from an optical image. Therefore, the information processing device 200 can more appropriately support the operator in determining whether the laterality of the subject is appropriate by using the determined subject information. In addition, the determined subject information can be used as additional information to be provided to the operator together with a live image in radiography for medical examination. Furthermore, the trained model used for estimating the skeletal information according to this embodiment is obtained by performing additional learning on a general-purpose trained model according to the purpose of each medical institution or medical site. Therefore, the information processing device 200 can appropriately estimate skeletal information required in a medical institution or medical site without collecting a large amount of training data.

[0076] The subject information determination unit 203 can determine the subject information by rule-based processing. Therefore, the information processing device 200 can determine the subject information by rule-based processing that is relatively easy to adjust, without adjusting a trained model whose configuration is difficult to change. Therefore, the information processing device 200 can determine the subject information by rules according to the operation and knowledge of a medical institution or a medical site.

[0077] Furthermore, the skeleton estimation unit 202 can estimate the coordinates of a plurality of feature points of the subject in the optical image and the probability that the plurality of feature points correspond to predefined feature points of the subject as skeleton information of the subject in the optical image. In this case, the subject information determination unit 203 can select the feature points of the subject in the optical image using a threshold value for the probability estimated by the skeleton estimation unit 202, and determine the subject information using the selected feature points. By selecting feature points in this way, the information processing device 200 can determine the subject information based on more appropriate feature points.

[0078] Furthermore, the subject information determination unit 203 can determine a radiation irradiation planned region (irradiation region) to be irradiated with radiation in the optical image. The subject information determination unit 203 can select feature points of the subject in the optical image based on the positional relationship between the coordinates estimated by the skeleton estimation unit 202 and the irradiation region, and determine subject information using the selected feature points. By selecting feature points in this way, the information processing device 200 can determine subject information based on more appropriate feature points.

[0079] The subject information determination unit 203 can determine the irradiation area based on the irradiation area of ​​the collimator lamp in the optical image or the center of the optical image. The subject information determination unit 203 can also determine the irradiation area based on the arrangement of the radiation generation device 120 that irradiates radiation and the camera 140 that functions as an example of an optical device that generates an optical image. In this case, by taking into account the arrangement of the radiation generation device 120 in the camera coordinate system, it is possible to perform appropriate analysis processing according to the position of the device for each medical site, and the analysis accuracy of the subject information determination unit 203 can be improved.

[0080] The skeleton estimation unit 202 can also extract a subject region in the optical image and estimate skeletal information of the subject in the optical image by using the subject region in the optical image as input data for the second trained model. The skeleton estimation unit 202 can also classify the optical image into classes indicating whether the subject's whole body, head, hands, or feet are included, and select and use the second trained model corresponding to the class. In these cases, the skeleton estimation unit 202 can be expected to estimate more appropriate skeletal information.

[0081] The second trained model may be a trained model obtained by additionally training using a smaller number of training data than the number of training data of the first trained model. For example, the second trained model may be a trained model obtained by additionally training the first trained model using skeletal information on at least one or more feature points of the left and right elbow joints, wrists, fingers, front and back of the hands, knee joints, ankles, toes, and front and back of the feet.

[0082] The information processing device 200 may further include a consistency determination unit 204 that functions as an example of a determination unit that determines whether or not there is consistency between the subject information and information included in the radiography order. In this case, the information processing device 200 can more appropriately support the operator in determining whether or not the radiography conditions, such as the subject's posture, correspond to the radiography order by presenting the determination result by the consistency determination unit 204.

[0083] In relation to this, the information processing device 200 may further include a display control unit 207 that causes the display unit 235 to display at least one of the optical image, the subject skeletal information in the optical image, the subject region in the optical image, the subject information, and the determination result by the consistency determination unit 204. Note that, when the consistency determination unit 204 outputs a determination result indicating that consistency is not achieved, the display control unit 207 can cause the display unit 235 to display a warning. In this case, the information processing device 200 can present the operator with the fact that consistency is not achieved and urge the operator to adjust the imaging conditions, such as the subject's posture, to correspond to the imaging order, thereby more appropriately supporting radiation imaging.

[0084] The information processing device 200 may further include an annotation unit 206 that arranges the subject information in the optical image or the radiation image as annotation information. In this case, the information processing device 200 can more appropriately support the operator in determining the imaging conditions such as the subject's posture by presenting the optical image or the radiation image on which the annotation information is arranged. The display control unit 207 can display the annotation information and the optical image or the radiation image on which the annotation information is arranged on the display unit 235.

[0085] Example 2 Hereinafter, a radiography system, an information processing device, and an information processing method according to a second embodiment of the present disclosure will be described with reference to Figs. 9 to 14. In this embodiment, a process of recognizing a part of a subject to be imaged from an optical image acquired at a predetermined frame rate using a video camera as the camera 140 and outputting it as subject information will be described. Here, the part of the subject includes, for example, the head, chest, and limbs of a human body. In addition, the granularity of the part to be recognized may vary depending on the purpose, so that, for example, the limbs can be divided into smaller parts such as the upper arm, elbow, wrist, hand, and finger. In particular, a case where the part to be imaged in an imaging order is determined to be either the "chest" or the "abdomen" will be described as an example.

[0086] Since the configurations of the radiation imaging system and the information processing device according to this embodiment are similar to those of the radiation imaging system and the information processing device according to the first embodiment, the same reference numerals are used and the description is omitted. Also, the description of the same processing as that described in detail in the first embodiment is omitted.

[0087] (Processing flow) A series of processing steps according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the processing steps according to this embodiment. When the processing steps according to this embodiment are started, the process proceeds to step S901.

[0088] (Step S901) In step S901, the optical image acquisition unit 201 controls the camera 140 to acquire optical images of the radiation imaging site including the subject of radiation imaging. In this embodiment, the camera 140 is a video camera attached to a radiation generator, and captures an image of the subject taking an imaging posture on the radiation detector 130 arranged on a supine table, and outputs optical images at a predetermined frame rate.

[0089] Here, a case where the radiation detector 130, a head 1002, a chest 1003, and an abdomen 1004 are captured in an optical image 1000 will be described with reference to Fig. 10. Fig. 10 shows an example of an optical image according to this embodiment. Here, a radiation detector region 1001 in the optical image 1000 is a region showing the radiation detector 130. Furthermore, in the example shown in Fig. 10, it is assumed that light from a collimator lamp is irradiated onto the subject's chest, and a collimator lamp irradiation region 1005 is depicted on the chest 1003 in the optical image 1000. The collimator lamp irradiation region 1005 coincides with a region to be irradiated with radiation during radiation imaging.

[0090] (Step S902) In step S902, the skeleton estimation unit 202 estimates skeleton information of the subject by using the optical image acquired by the optical image acquisition unit 201 as input data for the trained model. In this embodiment, a case will be described in which the skeleton estimation unit 202 applies whole-body skeleton estimation, which estimates left and right eyes, left and right shoulder joints, left and right hip joints, left and right elbows, left and right wrists, left and right knees, and left and right ankles as skeleton information, to the optical image 1000. As shown in Fig. 11, the skeleton estimation unit 202 detects left and right eyes 1101, 1102, left and right shoulder joints 1103, 1104, and left and right hip joints 1105, 1106 in the optical image 1000 as feature points.

[0091] The skeleton estimation unit 202 outputs the coordinates and probability of each feature point in a format as shown in Table 1100 as skeleton information. Here, it is assumed that the skeleton estimation unit 202 outputs the coordinates (x1101, y1101) to (x1106, y1106) of each feature point and the probability P1101 to P1106 indicating the likelihood of each feature point. However, it is assumed that the probability that other skeleton information is included in the optical image 1000 is 0.0, and the coordinates corresponding to each feature point are not output. Also, for the sake of simplicity, it is assumed that no erroneous estimation is made here, unlike the example of the first embodiment. Note that the skeleton estimation unit 202 may output skeleton information in a format according to the output of the trained model, similar to the skeleton estimation unit 202 according to the first embodiment.

[0092] In addition, the trained model used in this embodiment may be a trained model that has been additionally trained to a trained model used for general-purpose skeletal estimation so as to be able to output feature points required for each medical institution or medical site, similar to the trained model according to Example 1. The training data may also be prepared similarly to the training data according to Example 1.

[0093] (Step S903) In step S903, the subject information determination unit 203 determines a part of the subject to be imaged from the skeletal information estimated by the skeleton estimation unit 202, and outputs the part as subject information. Specifically, the subject information determination unit 203 applies a rule-based algorithm as shown in Fig. 12 to the skeletal information estimated by the skeleton estimation unit 202. In this rule-based algorithm, the part of the subject to be imaged is determined based on the probability and coordinates of the skeletal information output by the skeleton estimation unit 202.

[0094] Here, an example will be described in which the part of the subject is determined to be either the chest or the abdomen by applying the rule-based algorithm in Fig. 12 to the output of the skeleton estimation shown in Fig. 11. The output of the skeleton estimation shown in Fig. 11 includes left and right eyes 1101, 1102, left and right shoulder joints 1103, 1104, and left and right hip joints 1105, 1106, and each of the probabilities P1101 to P1106 is assumed to be equal to or greater than a predetermined threshold value.

[0095] First, in step S1201, the object information determination unit 203 deletes feature points whose probability of being a feature point is less than a threshold value from among the feature points included in the skeleton information output by the skeleton estimation unit 202. In the above example, since all of the probabilities P1101 to P1106 are equal to or greater than a predetermined threshold value, there are no feature points to be deleted as feature points below the threshold value.

[0096] Next, in step S1202, the object information determination unit 203 determines whether the number of remaining feature points is one or more. If it is determined in step S1202 that the number of feature points is more than one, the process proceeds to step S1203. On the other hand, if it is determined in step S1202 that the number of feature points is one, the process proceeds to step S1204. In the above example, the object information determination unit 203 determines that the number of remaining feature points is more than one, and therefore the process proceeds to step S1203.

[0097] In step S1203, the object information determination unit 203 obtains the position of the region to be irradiated with radiation in the optical image, and deletes, from the remaining feature points, those feature points other than the feature point closest to the position of the region to be irradiated with radiation. In the above example, the object information determination unit 203 calculates the distance between the position of the region to be irradiated with radiation and the coordinates of the feature points (x1101, y1101) to (x1106, y1106).

[0098] Here, the position of the region to be irradiated with radiation in the optical image 1000 may be the collimator lamp irradiation region 1005. Therefore, similar to the first embodiment, the subject information determining unit 203 may extract the collimator lamp irradiation region 1005 by threshold processing based on the luminance of the optical image or the like, and calculate the distance from the feature point by regarding the collimator lamp irradiation region 1005 as the region to be irradiated with radiation. Note that the region to be irradiated with radiation may be calculated in consideration of the center of the optical image and the spatial positions of the radiation generating device 120 and the camera 140, as described in the first embodiment.

[0099] The subject information determination unit 203 can compare the distance between each feature point and the region to be irradiated with radiation, find the feature point that is closest to the region to which radiation sickness is likely to occur, and delete feature points other than the feature point. In the above example, the left and right shoulder joints 1103 and 1104 are closest to the region to be irradiated with radiation among the outputs of the skeleton estimation shown in FIG. 11, so the subject information determination unit 203 deletes the other feature points.

[0100] In step S1204, the subject information determination unit 203 outputs the part of the remaining feature point as subject information. In the above example, the subject information determination unit 203 outputs the part corresponding to either the left or right shoulder joint 1103, 1104, which is the remaining feature point. Since the rule-based algorithm according to this embodiment is an algorithm for determining whether the part is the chest or the abdomen, the determination result is output as "chest" which is close to the shoulder joint. In the algorithm for determining whether the part is the chest or the abdomen, for example, even if the feature point closest to the planned radiation irradiation region is either the left or right eye, "chest" is output. On the other hand, "abdomen" is output when the feature point closest to the planned radiation irradiation region is the left or right hip joint 1105, 1106, or the left or right wrist or knee joint, which is not shown here. When the part of the subject is output in step S1204, the laterality determination process, which is the process for determining the subject information according to this embodiment, is completed.

[0101] The above describes a case where the feature points are the left and right eyes, shoulder joints, and hip joints, and the part of the subject is determined to be the chest or the abdomen. However, other feature points may be used, or other parts such as the head and limbs may be determined, as the processing of the subject information determination unit 203 that outputs the part as the subject information. The rule-based algorithm for determining the subject information may differ depending on the shooting posture adopted by the medical institution or medical site, the feature points captured in the angle of view of the camera 140 used, and the like. For example, in a rule-based algorithm that aims to determine the head and limbs as well, if the feature point closest to the radiation irradiation planned area is either the left or right eye, the "head" may be output. In addition, in a rule-based algorithm that aims to determine the left and right wrists and knee joints as well, the part may be output with a granularity such as the "left wrist", "right wrist", "left knee joint", and "right knee joint".

[0102] There are other possible rule-based algorithms for determining the part of the subject. For example, as shown in Fig. 13, it is assumed that a certain medical institution has a camera 140 installed so that an optical image for chest imaging is an optical image 1300, and an optical image for abdominal imaging is an optical image 1301. In this case, the subject information determination unit 203 may apply a rule-based algorithm as shown in Fig. 14 to the output of the bone structure estimation.

[0103] In the rule-based algorithm shown in FIG. 14, first, in step S1401, the object information determination unit 203 simply determines whether there is a feature point belonging to the head. If it is determined in step S1401 that there is a feature point belonging to the head, the process proceeds to step S1402. In step S1402, the object information determination unit 203 outputs "chest" as object information. On the other hand, if it is determined in step S1401 that there is no feature point belonging to the head, the process proceeds to step S1403. In step S1403, the object information determination unit 203 outputs "abdomen" as object information. When the part of the object is output in step S1402 or step S1403, the laterality determination process, which is the object information determination process according to this embodiment, is completed.

[0104] In this way, the rule-based algorithm for determining the subject information may correspond to the operation at the medical institution or medical site where the radiation imaging system 100 or the information processing device 200 is used. Here, the operation at the medical institution or medical site may include, for example, an agreement on the posture of the subject during radiation imaging and the arrangement of the camera 140 according to the imaging part.

[0105] (Example of use of determined specimen information) The subject information including the subject's part obtained as described above may be used for processing such as annotation, as in the first embodiment. For example, the subject information including the subject's part obtained may be transferred to the consistency determination unit 204 via the CPU bus 230 and used to determine the consistency between the imaging order information and the subject information. The display control unit 207 can also display the subject information and analysis information obtained during the process on the display unit 235. Furthermore, the annotation unit 206 can annotate the subject information on the acquired radiation image. The display control unit 207 can also display the radiation image annotated with the subject information on the display unit 235.

[0106] As described above, the subject information determination unit 203 can determine subject information including information indicating the subject's part in the optical image by using the subject's skeletal information in the optical image. With the above configuration, the information processing device 200 according to this embodiment can determine the part of the subject (radiography subject) as subject information by using the skeletal information estimated from the optical image. Therefore, the information processing device 200 can more appropriately support the operator's judgment regarding the suitability of the subject's part by using the determined subject information. Furthermore, similar to the first embodiment, the determined subject information can be used as additional information to be provided to the operator together with a live image in radiography for medical examination. Furthermore, similar to the first embodiment, the information processing device 200 can appropriately estimate skeletal information required in a medical institution or a medical site without collecting a large amount of learning data. Note that in this embodiment, the subject information including information indicating the subject's part is determined, but information indicating the laterality and part of the subject may be determined as the subject information.

[0107] Example 3 A radiation imaging system, an information processing device, and an information processing method according to a third embodiment of the present disclosure will be described below with reference to Fig. 15. In this embodiment, a customization (adjustment) process will be described for recognizing the laterality and part of a subject to be imaged from optical images acquired at a predetermined frame rate using a video camera as the camera 140, and outputting the result as subject information.

[0108] Since the configurations of the radiation imaging system and the information processing device according to this embodiment are similar to those of the radiation imaging system and the information processing device according to the first embodiment, the same reference numerals are used and the description is omitted. Also, the description of the same processing as that described in detail in the first and second embodiments is omitted.

[0109] (Processing flow) A processing procedure of an adjustment process of a rule-based algorithm related to the object information determination process according to this embodiment will be described with reference to Fig. 15. Fig. 15 is a flowchart showing the processing procedure according to this embodiment. When the adjustment process of the rule-based algorithm related to the object information determination process according to this embodiment is started, the process proceeds to step S1501.

[0110] (Step S1501) In step S1501, the optical image acquisition unit 201 controls the camera 140 to acquire an optical image of the radiation imaging site including the subject of radiation imaging. In this embodiment, the camera 140 is a video camera attached to a radiation generator, and captures an image of the subject taking an imaging posture on the radiation detector 130 arranged on a supine table, and outputs an optical image at a predetermined frame rate. Here, as in the second embodiment, a case in which the radiation detector 130, head 1002, chest 1003, and abdomen 1004 are captured in the optical image 1000 will be described as an example with reference to FIG. 10.

[0111] (Step S1502) In step S1502, the skeleton estimation unit 202 estimates skeleton information of the subject by using the optical image acquired by the optical image acquisition unit 201 as input data for the trained model. In this embodiment, a case will be described in which the skeleton estimation unit 202 applies whole-body skeleton estimation, which estimates left and right eyes, left and right shoulder joints, left and right hip joints, left and right elbows, left and right wrists, left and right knees, and left and right ankles as skeleton information, to the optical image 1000. As in the second embodiment, the skeleton estimation unit 202 can detect left and right eyes 1101, 1102, left and right shoulder joints 1103, 1104, and left and right hip joints 1105, 1106 in the optical image 1000 as feature points, as shown in FIG.

[0112] The trained model used in this embodiment may be a trained model that has been additionally trained to a trained model used for general-purpose skeletal estimation so as to output feature points required for each medical institution or medical site, similar to the trained model according to the first embodiment. The training data may also be prepared similarly to the training data according to the first embodiment.

[0113] (Step S1503) In step S1503, the subject information determination unit 203 adjusts parameters of a rule-based algorithm for determining a part of the subject that is to be captured as a radiographic image, in response to an input from the operator via the operation unit 234. More specifically, the display control unit 207 causes the display unit 235 to display the skeletal information estimated by the skeleton estimation unit 202. Based on the skeletal information displayed on the display unit 235, the operator instructs the information processing device 200 via the operation unit 234 as to the part of the subject that should be output as subject information. The subject information determination unit 203 adjusts various parameters of the rule-based algorithm as shown in FIG. 12 in response to an instruction from the operator.

[0114] Here, the parameters of the rule-based algorithm may include, for example, a threshold value for threshold processing applied to the probability in step S1201, and the number of feature points serving as a criterion for determining whether to branch the process in step S1202. Furthermore, the parameters of the rule-based algorithm may include, for example, the positional relationship between the region to be irradiated with radiation and the feature points to be left and the region to be irradiated with radiation in step S1203, and the correspondence relationship between the remaining feature points and the part to be output in step S1204. Note that the positional relationship between the feature points to be left and the region to be irradiated with radiation may include, for example, the order of proximity of the feature points to be left to the region to be irradiated with radiation (for example, closest or second closest, etc.).

[0115] For example, a case will be described in which the operator inputs an instruction via the operation unit 234 that the part determined from the estimated skeletal information is "chest" in order to adjust the rule-based algorithm as shown in FIG. 12. In this case, the object information determining unit 203 determines the planned radiation irradiation area and determines which of the left and right eyes 1101, 1102, the left and right shoulder joints 1103, 1104, and the left and right hip joints 1105, 1106 is the closest feature point to the planned radiation irradiation area. In the above example, either of the shoulder joints 1103, 1104 is closest. Therefore, when the shoulder joint is closest to the planned radiation irradiation area, the object information determining unit 203 adjusts the parameters of the rule-based algorithm so as to determine and output the object information as "chest". In this case, the object information determining unit 203 can adjust the positional relationship between the feature points to be left and the planned radiation irradiation area so as to determine either of the shoulder joints 1103, 1104 as the feature points to be left in step S1203. Furthermore, the object information determining unit 203 can adjust the correspondence between the remaining feature points in step S1204 and the part names to be output.

[0116] Furthermore, the subject information determination unit 203 may store, in the storage unit 232, coordinates of feature points with a probability equal to or higher than a threshold value among the skeletal information output by the skeleton estimation unit 202 together with the body part information input by the operator, without calculating the planned radiation irradiation region. In this case, information such as the coordinates of feature points when the body part information is registered as the chest and the coordinates of feature points when the body part information is registered as the abdomen is stored in the storage unit 232. In such a case, when inputting an optical image to determine the body part of the subject, the subject information determination unit 203 may obtain the sum of the distances between the coordinates of the feature points obtained by inputting the optical image and the coordinates of the stored feature points, and determine the body part with the smallest sum as the subject information.

[0117] Regarding the method of adjusting the parameters of the rule-based algorithm, a configuration may be adopted in which a plurality of parameter combinations are prepared in advance, and one of the parameter combinations is selected in response to an instruction from an operator. For example, the object information determination unit 203 can select a parameter combination by referring to a lookup table prepared in advance in response to an instruction from an operator, and adjust the parameters. The parameter combination may include, for example, a parameter combination related to a process as shown in FIG. 12, a parameter combination related to a process using coordinates of a feature point equal to or greater than a threshold, and a parameter combination related to a process as shown in FIG. 14. The parameter combination may also include, for example, a parameter combination related to a process for determining the laterality of the object as shown in FIG. 12. The instruction from the operator may specify object information to be output as described above, or may specify a parameter combination.

[0118] In step S1503, the parameters of the rule-based algorithm are adjusted, and the process ends.

[0119] In this embodiment, an example of adjusting the parameters of the rule-based algorithm for the process of determining the part of the subject as the subject information has been described. In contrast, for the process of determining the laterality of the subject as the subject information as in the first embodiment, the parameters of the rule-based algorithm can be adjusted in the same manner as in this embodiment. In this case, the parameters of the rule-based algorithm may include, for example, a threshold value for the threshold process applied to the probability in step S601, and the number of feature points serving as a criterion for determining whether to branch the process in step S602. In addition, the parameters of the rule-based algorithm may include a positional relationship between the region to be irradiated with radiation and the feature points to be left in step S603, and a correspondence relationship between the remaining feature points and the laterality to be output in step S604. In addition, for the process of determining the part and laterality of the subject as the subject information, the parameters of the rule-based algorithm can be adjusted in the same manner as in this embodiment.

[0120] As described above, the subject information determination unit 203 according to this embodiment can adjust the parameters of the rule-based processing based on the skeletal information of the subject in the optical image and the subject information acquired via the operation unit 234. Here, the parameters of the rule-based processing can include at least one of the threshold value of the threshold processing for selecting feature points of the subject in the optical image, the positional relationship between the feature points and the irradiation area to which radiation is irradiated, and the correspondence relationship between the feature points and at least one of the part and the laterality of the subject. With the above configuration, the information processing device 200 according to this embodiment can flexibly adapt the subject information determination process to the operation and knowledge of the medical institution or medical site by adjusting the rules according to the operation and knowledge of the medical institution or medical site.

[0121] In this embodiment, as in the first and second embodiments, the optical image acquisition unit 201 acquires an optical image of a radiation imaging site. On the other hand, the optical image used in the adjustment process of the rule-based algorithm may be an optical image acquired at a virtual radiation imaging site by imaging a human body phantom. That is, the subject information determination unit 203 can adjust the parameters of the rule-based process based on the skeletal information acquired using the optical image acquired by imaging the phantom and the subject information acquired via the operation unit 234. In this case, an image that makes it easier to estimate the skeleton in step S1502 can be acquired than when an actual subject is used, and the parameters can be adjusted more appropriately in step S1503.

[0122] In addition, the optical image used in the adjustment process of the rule-based algorithm does not have to be obtained by actually capturing an image. For example, a two-dimensional image on which three-dimensional coordinates representing a virtual skeletal position calculated from virtual object data using a three-dimensional modeling tool are projected may be used. In this case, the object information determination unit 203 may use the skeletal information output by the skeleton estimation unit 202 based on the two-dimensional image for the parameter adjustment process. That is, the object information determination unit 203 can adjust the parameters of the rule-based process based on the skeletal information obtained by projecting the three-dimensional coordinates of the virtual object data generated by the three-dimensional modeling tool onto the two-dimensional image coordinates and the object information acquired via the operation unit 234. According to this method, an image showing the skeletal position in a desired posture can be relatively easily acquired on the information processing device 200 without placing the human body or phantom in an arbitrary posture. Note that such a two-dimensional image may be used as input data for learning data related to additional learning of the learned model used by the skeleton estimation unit 202.

[0123] In this embodiment, the subject information determination unit 203 adjusts the parameters of the rule-based algorithm related to the subject information determination process in response to an instruction from the operator. In contrast, the information processing device 200 may be provided with a programming environment, and the operator may be able to describe the handling of the skeletal information output by the skeleton estimation unit 202 as a program via the operation unit 234. Such a configuration may use a programming language such as C language or Python. In addition, the information processing device 200 may be configured to use a no-code development platform that can build a rule-based algorithm using a graphical user interface. Furthermore, in recent years, it has become possible to generate a program by inputting instructions in natural language using an artificial intelligence chatbot, which is a type of generative AI. Therefore, the information processing device 200 may be configured to utilize a generative AI such as an artificial intelligence chatbot.

[0124] Furthermore, the learning data of the trained model is not limited to data obtained using the camera 140 itself that actually captures images, and may be data obtained using the same type of camera or the same kind of camera, depending on the desired configuration. Note that the trained model for skeletal estimation according to the above-mentioned embodiment and examples is considered to extract, for example, the magnitude of the luminance value of the optical image, the order and inclination of the bright and dark areas, position, distribution, continuity, etc. as part of the feature amount and use them in the estimation process of skeletal information.

[0125] The trained model for skeletal estimation described above can be provided in the information processing device 200. The inference device (trained model) may be configured, for example, by a software module executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured by a circuit that performs a specific function such as an ASIC. The inference device may be provided in another device such as a server connected to the information processing device 200. In this case, the information processing device 200 can use the inference device by connecting to a server or the like that includes the inference device via any network such as the Internet. Here, the server that includes the inference device may be, for example, a cloud server, a fog server, or an edge server. In addition, when configuring a network within a facility, a site that includes a facility, or an area that includes multiple facilities to be capable of wireless communication, the reliability of the network may be improved by configuring the network to use radio waves in a dedicated wavelength band that is limited to the facility, site, area, or the like. In addition, the network may be configured by wireless communication that allows high-speed communication, large-capacity communication, low-latency communication, and multiple simultaneous connections.

[0126] (Other Examples) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments and examples is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions. A computer may have one or more processors or circuits, and may include separate computers or a network of separate processors or circuits for reading and executing computer-executable instructions.

[0127] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA), and the processor or circuitry may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0128] The above disclosure includes the following configurations, methods, and programs. (Configuration 1) an acquisition unit that acquires an optical image obtained by imaging a subject at a radiation image capturing site; an estimation unit that estimates skeletal information of a subject in an optical image by using the optical image as input data of a second trained model obtained by additionally learning a first trained model that estimates skeletal information of a subject's skeleton using skeletal information regarding a skeleton different from a skeleton indicated by skeletal information learned by the first trained model; a determination unit that determines object information including at least one of information on a side and a part of the object in the optical image, using skeletal information of the object in the optical image; An information processing device comprising: (Configuration 2) 2. The information processing apparatus according to configuration 1, wherein the determination unit determines the object information by rule-based processing. (Configuration 3) 3. The information processing device according to configuration 1 or 2, wherein the estimation unit estimates coordinates of a plurality of feature points of the subject in the optical image and a probability that the plurality of feature points correspond to predefined feature points of the subject, as skeletal information of the subject in the optical image. (Configuration 4) The information processing device according to configuration 3, wherein the determination unit selects feature points of the object in the optical image by using a threshold value for the probability estimated by the estimation unit, and determines the object information by using the selected feature points. (Configuration 5) The determination unit is determining an irradiation area in the optical image where radiation is irradiated; selecting feature points of the object in the optical image based on a positional relationship between the coordinates estimated by the estimation unit and the irradiation region; 5. The information processing device according to configuration 3 or 4, wherein the object information is determined using the selected feature points. (Configuration 6) The information processing device according to configuration 5, wherein the determination unit determines the irradiation area based on an irradiation area of ​​a collimator lamp in the optical image or a center of the optical image. (Configuration 7) 6. The information processing device according to configuration 5, wherein the determination unit determines the irradiation region based on an arrangement of a radiation generation device that irradiates the radiation and an optical device that generates the optical image. (Configuration 8) The information processing device of any one of configurations 1 to 7, wherein the estimation unit extracts an object region in the optical image, and estimates skeletal information of the object in the optical image by using the object region in the optical image as input data of the second trained model. (Configuration 9) The information processing device according to any one of configurations 1 to 8, wherein the estimation unit classifies the optical image into a class indicating whether the optical image includes the entire body, a head, a hand, or a foot of the subject, and selects and uses a second trained model corresponding to the class. (Configuration 10) The information processing device according to any one of configurations 1 to 9, wherein the second trained model is a trained model obtained by performing additional training using a number of training data smaller than the number of training data of the first trained model. (Configuration 11) The information processing device according to any one of configurations 1 to 10, wherein the second trained model is a trained model obtained by additionally training the first trained model using skeletal information regarding at least one or more feature points of left and right elbow joints, wrists, fingers, front and back of hands, knee joints, ankles, toes, and front and back of feet. (Configuration 12) 12. The information processing device according to any one of configurations 1 to 11, further comprising a determination unit that determines whether or not there is consistency between the subject information and information included in an order for capturing the radiation image. (Configuration 13) 13. The information processing device according to configuration 12, further comprising a display control unit that causes a display unit to display at least one of the optical image, skeletal information of the subject in the optical image, the subject region in the optical image, the subject information, and the determination result by the determination unit. (Configuration 14) 13. The information processing device according to configuration 12, further comprising a display control unit that displays a warning on a display unit when the determination unit outputs a determination result indicating that the consistency is not achieved. (Configuration 15) 13. The information processing device according to any one of configurations 1 to 12, further comprising an annotation unit that arranges the object information in the optical image or the radiation image as annotation information. (Configuration 16) The information processing device of configuration 15 further includes a display control unit that causes a display unit to display at least one of the optical image, skeletal information of the subject in the optical image, the subject region in the optical image, the subject information, the annotation information, and an optical image or a radiation image in which the annotation information is arranged. (Configuration 17) The information processing device according to configuration 2, wherein the determination unit adjusts parameters of the rule-based processing based on skeletal information of the subject in the optical image, skeletal information obtained using an optical image obtained by photographing a phantom, or skeletal information obtained by projecting three-dimensional coordinates of virtual subject data generated by a three-dimensional modeling tool onto two-dimensional image coordinates, and subject information acquired via an operation unit. (Configuration 18) 18. The information processing device according to claim 17, wherein the parameters include at least one of a threshold for threshold processing related to selecting feature points of the subject in the optical image, a positional relationship between the feature points and an irradiation area to which radiation is irradiated, and a correspondence relationship between the feature points and at least one of a part and a side of the subject. (Configuration 19) An optical device for taking an optical image of a subject; a radiation generating device and a radiation detector for performing radiation imaging of the subject; An information processing device according to any one of configurations 1 to 18; A radiation imaging system comprising: (Method 1) Obtaining an optical image by imaging a subject at a radiation imaging site; Estimating skeletal information of the subject in the optical image by using the optical image as input data of a second trained model obtained by additionally training a first trained model that estimates skeletal information about the subject's skeleton using skeletal information about a skeleton different from the skeleton indicated by the skeletal information learned by the first trained model; determining object information including at least one of information on a side and a part of the object in the optical image using skeletal information of the object in the optical image; An information processing method comprising: (Program 1) A program that, when executed by a computer, causes the computer to execute each step of the information processing method described in Method 1.

[0129] Although the present invention has been described above with reference to the embodiments and examples, the present invention is not limited to the above embodiments and examples. The present invention also includes inventions that have been modified within the scope of the present invention and inventions equivalent to the present invention. In addition, the above-mentioned embodiments and examples can be appropriately combined within the scope of the present invention. [Explanation of symbols]

[0130] 200: information processing device, 201: optical image acquisition unit (acquisition unit), 202: bone structure estimation unit (estimation unit), 203: subject information determination unit (determination unit)

Claims

1. an acquisition unit that acquires an optical image obtained by imaging a subject at a radiation image capturing site; an estimation unit that estimates skeletal information of a subject in an optical image by using the optical image as input data of a second trained model obtained by additionally learning a first trained model that estimates skeletal information of a subject's skeleton using skeletal information regarding a skeleton different from a skeleton indicated by skeletal information learned by the first trained model; a determination unit that determines object information including at least one of information on a side and a part of the object in the optical image, using skeletal information of the object in the optical image; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the determination unit determines the object information by rule-based processing.

3. 2. The information processing device according to claim 1, wherein the estimation unit estimates coordinates of a plurality of feature points of the subject in the optical image and a probability that the plurality of feature points correspond to predefined feature points of the subject, as skeletal information of the subject in the optical image.

4. The information processing device according to claim 3 , wherein the determination unit selects feature points of the object in the optical image by using a threshold value for the probability estimated by the estimation unit, and determines the object information by using the selected feature points.

5. The determination unit is determining an irradiation area in the optical image where radiation is irradiated; selecting feature points of the object in the optical image based on a positional relationship between the coordinates estimated by the estimation unit and the irradiation region; The information processing apparatus according to claim 3 , wherein the object information is determined using the selected feature points.

6. The information processing apparatus according to claim 5 , wherein the determination unit determines the illumination area based on an illumination area of ​​a collimator lamp in the optical image or a center of the optical image.

7. The information processing apparatus according to claim 5 , wherein the determination unit determines the irradiation region based on an arrangement of a radiation generation device that irradiates the radiation and an optical device that generates the optical image.

8. The information processing device according to claim 1 , wherein the estimation unit extracts an object region in the optical image, and estimates skeletal information of the object in the optical image by using the object region in the optical image as input data for the second trained model.

9. 2. The information processing device according to claim 1, wherein the estimation unit classifies the optical image into a class indicating whether the optical image includes the entire body, a head, a hand, or a foot of the subject, and selects and uses a second trained model corresponding to the class.

10. The information processing device according to claim 1 , wherein the second trained model is a trained model obtained by additional training using a number of training data smaller than the number of training data of the first trained model.

11. The information processing device according to claim 1, wherein the second trained model is a trained model obtained by additionally learning the first trained model using skeletal information regarding at least one or more feature points of left and right elbow joints, wrists, fingers, the front and back of the hands, knee joints, ankles, toes, and the front and back of the feet.

12. The information processing apparatus according to claim 1 , further comprising a determination unit that determines whether or not there is consistency between the subject information and information included in an order for capturing the radiation image.

13. The information processing device according to claim 12 , further comprising a display control unit that causes a display unit to display at least one of the optical image, skeletal information of the subject in the optical image, the subject region in the optical image, the subject information, and the determination result by the determination unit.

14. The information processing apparatus according to claim 12 , further comprising a display control unit that displays a warning on a display unit when the determination unit outputs a determination result indicating that the consistency is not achieved.

15. The information processing apparatus according to claim 1 , further comprising an annotation unit that arranges the object information in the optical image or the radiation image as annotation information.

16. The information processing device according to claim 15, further comprising a display control unit that causes a display unit to display at least one of the optical image, skeletal information of the subject in the optical image, the subject region in the optical image, the subject information, the annotation information, and an optical image or a radiation image on which the annotation information is arranged.

17. 3. The information processing device according to claim 2, wherein the determination unit adjusts parameters of the rule-based processing based on skeletal information of the subject in the optical image, skeletal information obtained using an optical image obtained by photographing a phantom, or skeletal information obtained by projecting three-dimensional coordinates of virtual subject data generated by a three-dimensional modeling tool onto two-dimensional image coordinates, and subject information acquired via an operation unit.

18. 18. The information processing device according to claim 17, wherein the parameters include at least one of a threshold value for threshold processing related to selecting feature points of the subject in the optical image, a positional relationship between the feature points and an irradiation region to which radiation is irradiated, and a correspondence relationship between the feature points and at least one of a part and a laterality of the subject.

19. An optical device for taking an optical image of a subject; a radiation generating device and a radiation detector for performing radiation imaging of the subject; An information processing device according to any one of claims 1 to 18; A radiation imaging system comprising:

20. Obtaining an optical image by imaging a subject at a radiation imaging site; Estimating skeletal information of the subject in the optical image by using the optical image as input data of a second trained model obtained by additionally training a first trained model that estimates skeletal information about the subject's skeleton using skeletal information about a skeleton different from the skeleton indicated by the skeletal information learned by the first trained model; determining object information including at least one of information on a side and a part of the object in the optical image using skeletal information of the object in the optical image; An information processing method comprising:

21. A program which, when executed by a computer, causes the computer to execute each step of the information processing method according to claim 20.

Images