Information processing device, radiation imaging system, information processing method, and program

Abstract

To provide an information processing device capable of acquiring subject information from an optical image using a learned model for estimating desired skeleton information, which is obtained by using learning data in the reduced number of pieces as compared with prior arts.SOLUTION: An information processing device includes: an acquisition unit for acquiring an optical image obtained by imaging a subject in an imaging site for a radiographic image: an estimation unit for estimating skeleton information on the subject in the optical image by using the optical image as input data for a second learned model acquired by performing additional learning using skeleton information on a skeleton different from the skeleton indicated by the skeleton information learned by a first learned model for the first learned model for estimating skeleton information on the skeleton of the subject; and a determination unit for determining the subject information including the information on at least one of the laterality and the region of the subject in the optical image, using information on the subject in the optical image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a radiation imaging system, an information processing method, and a program. [Background technology]

[0002] In recent years, in radiography for medical examinations, radiography support using optical images has become common, in which the radiography site is optically photographed to acquire an optical image, and additional information obtained by analyzing the optical image is provided to the operator along with a live image. For example, Patent Document 1 proposes a technology that determines the radiography position of the subject from an optical image and outputs information regarding the appropriateness of the radiography position, thereby enabling efficient radiography that does not depend on the skill or experience of the radiologist. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-199163 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 determines the imaging position of a subject from an optical image. However, the image analysis technology applied to such imaging position determination faces the challenge of further and continuous improvement of accuracy for a wider range of subjects and imaging conditions.

[0005] In general, in image processing technology, it is known that machine learning algorithms using large amounts of training data can create trained models that enable highly accurate recognition and classification.However, it is difficult to collect large amounts of training data, including optical images containing personal patient information, from radiography sites for medical examinations.

[0006] Therefore, one object of one embodiment of the present disclosure is to provide an information processing device that can acquire subject information from optical images using a trained model that estimates desired skeletal information, obtained using a reduced number of training data compared to conventional methods. [Means for solving the problem]

[0007] An information processing device according to an embodiment of the present disclosure includes an acquisition unit that acquires optical images obtained by imaging a subject at a radiographic imaging site; an estimation unit that estimates skeletal information of the subject in the optical images by using the optical images as input data of a second trained model obtained by additionally learning a first trained model that estimates skeletal information about the subject's skeleton using skeletal information about a skeleton different from the skeleton indicated by the skeletal information learned by the first trained model; and an estimation unit that estimates subject information including at least one of laterality and location of the subject in the optical images using the skeletal information of the subject in the optical images. , through rule-based processing and a determination unit that determines The estimation unit estimates coordinates of a plurality of feature points of the subject in the optical image and probabilities that the plurality of feature points correspond to predefined feature points of the subject as skeletal information of the subject in the optical image, and the determination unit obtains an irradiation region in the optical image to be irradiated with radiation, selects feature points of the subject in the optical image based on a positional relationship between the coordinates estimated by the estimation unit and the irradiation region, and determines the subject information using the selected feature points. . [Effects of the Invention]

[0008] According to one embodiment of the present disclosure, object information can be obtained from optical images using a trained model that estimates desired skeletal information, which is obtained using a reduced number of training data compared to conventional methods. [Brief explanation of the drawings]

[0009] [Figure 1] 1 shows a schematic configuration of a radiation imaging system according to an embodiment. [Figure 2] 1 shows a schematic configuration of an information processing apparatus according to an embodiment. [Figure 3] 1 is a flowchart illustrating a processing procedure according to the first embodiment. [Figure 4] 3A to 3C are diagrams illustrating examples of optical images according to the first embodiment. [Figure 5] 10A and 10B are diagrams illustrating an example of an output from a skeleton estimation unit according to the first embodiment. [Figure 6]10 is a flowchart illustrating an example of a subject information determination process according to the first embodiment. [Figure 7] FIG. 2 is a diagram illustrating the arrangement of a radiation generating device in a camera coordinate system. [Figure 8] 10 is a flowchart of an example of processing using object information according to the first embodiment. [Figure 9] 10 is a flowchart showing a processing procedure according to the second embodiment. [Figure 10] 10A and 10B are diagrams illustrating examples of optical images according to the second embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of an output from a skeleton estimation unit according to the second embodiment. [Figure 12] 10 is a flowchart illustrating an example of a subject information determination process according to the second embodiment. [Figure 13] 10A and 10B are diagrams illustrating another example of an optical image according to the second embodiment. [Figure 14] 10 is a flowchart showing another example of the object information determination process according to the second embodiment. [Figure 15] 10 is a flowchart showing a processing procedure according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, exemplary embodiments and examples for carrying out the present disclosure will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative positions of components, etc. described in the following embodiments and examples are arbitrary and can be changed depending on the configuration of the device to which the present invention is applied or various conditions. In addition, the same reference numerals are used between drawings to indicate identical or functionally similar elements.

[0011] In the following, the term radiation can include, for example, electromagnetic radiation such as X-rays and gamma rays, as well as particle radiation such as alpha rays, beta rays, particle rays, proton rays, heavy ion rays, and meson rays.

[0012] Furthermore, a machine learning model refers to a learning model based on a machine learning algorithm. Specific examples of machine learning algorithms include nearest neighbor algorithms, naive Bayes algorithms, decision trees, and support vector machines. Another example is deep learning, which uses a neural network to generate features and connection weighting coefficients for learning. Algorithms using decision trees include gradient boosting techniques such as LightGBM and XGBoost. Any of the above algorithms can be used as appropriate and applied to the following embodiments and examples. Furthermore, training data refers to training data and is composed of pairs of input data and output data. Furthermore, output data from training data is also referred to as ground truth data.

[0013] Furthermore, a trained model refers to a machine learning model that follows any machine learning algorithm, such as deep learning, and that has been trained (learned) in advance using appropriate training data (learning data). However, although a trained model is obtained in advance using appropriate training data, it does not mean that it does not undergo further learning, and additional learning can also be performed. Additional learning can also be performed after the device is installed at the site of use.

[0014] (Embodiment) (Schematic configuration) First, a radiation imaging system, an information processing device, and an information processing method according to an embodiment of the present disclosure will be described with reference to FIGS. 1 and 2. The embodiment of the present disclosure is applied to, for example, a radiation imaging system 100 and an information processing device 200 as shown in FIGS. 1 and 2. FIG. 1 shows a schematic configuration of the radiation imaging system 100 according to an embodiment of the present disclosure, and FIG. 2 shows a schematic configuration of the information processing device 200 according to an embodiment of the present disclosure. Note that while FIG. 1 shows a state in which the subject O is in a supine position, the subject O may be, for example, in an upright position or a sitting position. Furthermore, the imaging table used to support the subject O may be a table according to the position of the subject O.

[0015] The radiation imaging system 100 is provided with an information processing device 200, a radiation generation device 120, a radiation detector 130, and a camera 140. The information processing device 200 is connected to the radiation generation device 120, the radiation detector 130, and the camera 140 and can control them. The information processing device 200 can also perform image processing and analysis of various images obtained using the radiation detector 130 and the camera 140. The information processing device 200 is also connected to an external storage device 160 such as a server via an arbitrary network 150 such as the Internet or an intranet, and can exchange data with the external storage device 160. The external storage device 160 may also be directly connected to the information processing device 200.

[0016] The radiation generating device 120 includes, for example, a radiation generator such as a radiation tube, a collimator, a collimator lamp, etc., and can irradiate a radiation beam under the control of the information processing device 200. The radiation beam irradiated from the radiation generating device 120 passes through the object O while being attenuated, and enters the radiation detector 130.

[0017] The radiation detector 130 can detect an incident radiation beam and transmit a signal corresponding to the detected radiation beam to the information processing device 200. The radiation detector 130 may be any radiation detector that detects radiation and outputs a corresponding signal, and may be configured using, for example, an FPD (Flat Panel Detector). The radiation detector 130 may be an indirect conversion type detector that converts radiation into visible light using a scintillator or the like and then converts the visible light into an electrical signal using a photosensor or the like, or may be a direct conversion type detector that directly converts incident radiation into an electrical signal.

[0018] The camera 140 is an example of an optical device that captures an optical image of the subject O under the control of the information processing device 200 and acquires the optical image. The camera 140 transmits the optical image acquired by capturing the image to the information processing device 200. The camera 140 may have any known configuration, and may be configured as a camera capable of capturing moving images such as a video camera, or may be configured as a camera that only captures still images. The camera 140 may be configured to capture images using visible light, or may be configured to capture images using invisible light other than radiation, such as infrared light.

[0019] The information processing device 200 is provided with an optical image acquisition unit 201, a skeleton estimation unit 202, a subject information determination unit 203, a consistency determination unit 204, a radiographic image acquisition unit 205, an annotation unit 206, and a display control unit 207. The information processing device 200 also is provided with a CPU 231, a storage unit 232, a main memory 233, an operation unit 234, and a display unit 235. The various units of the information processing device 200 are connected via a CPU bus 230, and are capable of exchanging data with one another.

[0020] The optical image acquiring unit 201 can control the camera 140 and acquire an optical image of the subject O captured by the camera 140. The optical image acquiring unit 201 may also acquire an optical image of the subject O from the external storage device 160 or an optical device (not shown) connected to the information processing device 200 via an arbitrary network. The optical image acquiring unit 201 may also acquire an optical image stored in the storage unit 232.

[0021] The skeleton estimation unit 202 performs skeleton estimation using the optical image as input data for a trained model, and can estimate skeletal information of the human body in the optical image. The trained model used by the skeleton estimation unit 202 according to this embodiment can be a trained model generated by additionally training desired data on a general-purpose trained model for estimating skeletal information obtained using a large amount of training data. Here, the desired data may include skeletal information desired in a medical institution, a medical setting, or the like where the radiation imaging system 100 is used. Detailed processing by the skeleton estimation unit 202 will be described later.

[0022] The subject information determination unit 203 analyzes the estimated skeletal information and can determine and recognize subject information including information indicating the part of the subject O that is the subject of radiographic imaging in the optical image, information indicating the laterality, information indicating the direction, etc. Detailed processing by the subject information determination unit 203 will be described later.

[0023] The consistency determination unit 204 can determine whether there is consistency between the subject information determined by the subject information determination unit 203 and information about the subject O included in the radiographic image capturing order. The information processing device 200 can provide the operator with the determination result to assist in determining whether the subject information obtained using the optical image matches the radiographic imaging instructions.

[0024] The radiation image acquisition unit 205 controls the radiation generation device 120 and the radiation detector 130 to perform radiography of the subject O and acquire a radiation image of the subject O from the radiation detector 130. The radiation image acquisition unit 205 may also acquire a radiation image of the subject O from the external storage device 160 or a radiation detector (not shown) connected to the information processing device 200 via an arbitrary network. The radiation image acquisition unit 205 may also acquire a radiation image stored in the storage unit 232.

[0025] The annotation unit 206 annotates the object information determined by the object information determination unit 203 onto the radiation image. Here, annotation refers to a process of embedding object information indicating the part, laterality, direction, etc. of the object O into the radiation image. The annotation unit 206 may be configured to be able to annotate the object information onto the optical image.

[0026] The display control unit 207 can control the display of the display unit 235. The display control unit 207 can display, for example, patient information about the patient who is the subject O, imaging conditions, parameters set by the operator, generated optical images and radiation images, determined subject information, analysis information, and the like on the display unit 235. Here, the analysis information may include, for example, segmentation information, and the like. Furthermore, the display control unit 207 can display, on the display unit 235, any display or GUI, such as a button or slider for receiving an operation by the operator, according to a desired configuration.

[0027] The CPU (Central Processing Unit) 231 is an example of a processor that controls the operation of the information processing device 200. The CPU 231 uses the main memory 233 to control the operation of the entire device in accordance with operations from the operation unit 234 and parameters stored in the storage unit 232. Note that the processor in the information processing device 200 is not limited to a CPU, and may include, for example, a microprocessing unit (MPU) and a graphics processing unit (GPU). The processor may also include a digital signal processor (DSP), a data flow processor (DFP), and a neural processing unit (NPU).

[0028] The storage unit 232 can store various images, data, etc. processed by the information processing device 200. The storage unit 232 can also store patient information, imaging conditions, parameters set by the operator, etc. The storage unit 232 can also store information on a rule-based algorithm for analyzing skeletal information performed by the subject information determination unit 203. The storage unit 232 may be configured with any storage medium, such as an optical disk or memory. The main memory 233 is configured with memory, etc., and can be used for temporary data storage, etc.

[0029] Note that a GPU can perform efficient calculations by processing a larger amount of data in parallel. Therefore, when performing learning multiple times using a machine learning algorithm such as deep learning, it is effective to use a GPU for processing. Therefore, in this embodiment, a GPU may be used in addition to a CPU for processing by the information processing device 200, which functions as an example of a learning unit. Specifically, when executing a learning program including a learning model, learning can be performed by the CPU and GPU working together to perform calculations. Note that calculations may be performed only by the CPU or the GPU in the processing of the learning unit. Furthermore, the estimation process according to this embodiment may also be realized using a GPU, as with the learning unit. Note that if the trained model is provided in an external device, the information processing device 200 does not need to function as a learning unit.

[0030] The learning unit may also include an error detection unit and an update unit (not shown). The error detection unit obtains the error between correct data and output data output from the output layer of the neural network in response to input data input to the input layer. The error detection unit may use a loss function to calculate the error between the output data from the neural network and correct data. The update unit updates the connection weighting coefficients between the nodes of the neural network based on the error obtained by the error detection unit so as to reduce the error. This update unit updates the connection weighting coefficients using, for example, an error backpropagation method. The error backpropagation method is a technique for adjusting the connection weighting coefficients between the nodes of each neural network so as to reduce the error.

[0031] As the machine learning model according to this embodiment, for example, FCN (Fully Convolutional Network) or SegNet can be used. As the machine learning model for object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, as the machine learning model for object recognition in units of regions, YOLO (You Only Look Once), SSD (Single Shot Detector, or Single Shot MultiBox Detector) can be used.

[0032] The operation unit 234 includes input devices, such as a keyboard and a mouse, for operating the information processing device 200. The operator can also input parameters and the like related to a rule-based algorithm for analysis processing using skeletal information performed by the subject information determination unit 203 via the operation unit 234.

[0033] The display unit 235 includes, for example, any display, and displays various information such as subject information and various images under the control of the display control unit 207. The display unit 235 may be, for example, a monitor of a console for operating a radiation imaging apparatus including the radiation generation device 120 and the radiation detector 130. The display unit 235 may also be a sub-monitor installed at a position where the operator can observe while assisting in positioning the subject O, or a console monitor of a radiation irradiator. The display unit 235 may also be a device that allows the operator to reliably check the display with minimal eye movement, such as a head-mounted display that the operator can wear while working. The display unit 235 may also be configured as a touch panel display, in which case the display unit 235 can also serve as the operation unit 234.

[0034] The information processing device 200 can be configured by a computer provided with a processor and a memory. The information processing device 200 may be configured by a general computer or a computer dedicated to a radiation imaging system. The information processing device 200 may be, for example, a personal computer (PC), or a desktop PC, a notebook PC, or a tablet PC (portable information terminal). Furthermore, the information processing device 200 may be configured as a cloud-based computer in which some of the components are located in an external device.

[0035] Furthermore, the optical image acquisition unit 201, the skeleton estimation unit 202, the subject information determination unit 203, the consistency determination unit 204, the radiation image acquisition unit 205, the annotation unit 206, and the display control unit 207 may be configured as software modules executed by the CPU 231. Furthermore, each of these components may be configured as a circuit that performs a specific function, such as an ASIC, or an independent device.

[0036] Next, the operation of the information processing device 200 under the control of the CPU 231 will be described. First, based on radiography order information transmitted from an information management device (not shown), the information processing device 200 starts preparation for radiography, and the optical image acquisition unit 201 starts acquiring optical images under the control of the CPU 231. Here, the radiography order information is information corresponding to the unit of examination ordered by a doctor, and includes, for example, patient information, the (scheduled) radiography date and time, and the part, direction, and posture of the subject to be radiographed based on the doctor's findings. The radiography order information includes information necessary for radiography, such as the type of radiation detection device to be used (for upright position, for supine position, portable, etc.), the patient's posture (radiography part, direction, etc.), and radiography conditions (tube voltage, tube current, presence or absence of a grid, etc.).

[0037] The optical image acquisition unit 201 controls the camera 140 to capture an optical image of the subject O and acquires the optical image from the camera 140. The optical image acquired by the optical image acquisition unit 201 is transferred sequentially via the CPU bus 230 to the main memory 233, the skeleton estimation unit 202, and the subject information determination unit 203.

[0038] The skeleton estimation unit 202 estimates skeletal information of the subject O using the transferred optical images as input data for the trained model. Subsequently, the subject information determination unit 203 obtains subject information from the estimated skeletal information. The subject information is transferred to the consistency determination unit 204 via the CPU bus 230. The consistency determination unit 204 compares the imaging order information with the subject information and outputs a consistency determination result. The optical images, skeletal information, subject information, and consistency determination result are transferred to the storage unit 232 and display control unit 207 via the CPU bus 230. The storage unit 232 stores the transferred various pieces of information. The display control unit 207 displays the transferred various pieces of information on the display unit 235.

[0039] The operator checks the various types of displayed information and issues operation instructions as necessary via the operation unit 234. For example, if the consistency determination result is correct, the operator issues an instruction to capture a radiographic image via the operation unit 234. This instruction to capture a radiographic image is transmitted to the radiographic image acquisition unit 205 by the CPU 231.

[0040] Upon receiving an imaging instruction, the radiation image acquisition unit 205 controls the radiation generation device 120 and the radiation detector 130 to perform radiation imaging. In radiation imaging, first, a radiation beam is irradiated from the radiation generation device 120 toward the subject O, and the radiation beam that passes through the subject O while attenuating is detected by the radiation detector 130. The radiation image acquisition unit 205 acquires a signal corresponding to the intensity of the radiation beam detected by the radiation detector 130 as a radiation image. Data of this radiation image is transferred sequentially to the main memory 233 and the annotation unit 206 via the CPU bus 230.

[0041] The annotation unit 206 annotates the transferred radiographic image with the subject information stored in the storage unit 232. The annotated radiographic image is transferred to the storage unit 232 and the display control unit 207 via the CPU bus 230. The storage unit 232 stores the transferred annotated radiographic image. The display control unit 207 displays the transferred annotated radiographic image on the display unit 235. The operator can check the displayed radiographic image and issue operation instructions via the operation unit 234 as necessary.

[0042] Example 1 3 to 7, a radiography system, an information processing device, and an information processing method according to a first embodiment of the present disclosure will be described. In this embodiment, a process will be described in which the laterality of a subject to be imaged is recognized from optical images acquired at a predetermined frame rate using a video camera as the camera 140, and the processed images are output as subject information. Laterality is information indicating whether a body part to be imaged is on the left or right side.

[0043] (Processing flow) A series of processing steps according to this embodiment will be described below with reference to Fig. 3. Fig. 3 is a flowchart showing the processing steps according to this embodiment. When the processing steps according to this embodiment start, the process proceeds to step S301.

[0044] (Step S301) In step S301, the optical image acquisition unit 201 controls the camera 140 to acquire optical images of the radiation imaging site, including the subject being radiographed. In this embodiment, the camera 140 is a video camera attached to a radiation generator, which captures an image of the subject in a radiography position on the radiation detector 130 arranged on a supine table, and outputs optical images at a predetermined frame rate.

[0045] Here, with reference to FIG. 4, an example will be described in which the radiation detector 130, a right hand 402, and a left hand 403 are captured in an optical image 400. FIG. 4 shows an example of an optical image according to this embodiment. Here, a radiation detector region 401 in the optical image 400 is a region representing the radiation detector 130. Furthermore, in the example shown in FIG. 4, light from a collimator lamp is irradiated onto the subject's right hand, and a collimator lamp irradiation region 404 is depicted above the right hand 402 in the optical image 400. Here, the collimator lamp is a device that irradiates visible light and is attached to a collimator in order to confirm a radiation irradiation region, which is a region to be irradiated with radiation, before radiation irradiation. The collimator lamp irradiation region 404 generated by the collimator lamp coincides with the radiation irradiation region during radiography.

[0046] (Step S302) In step S302, the skeleton estimation unit 202 estimates skeletal information of the subject by using the optical image acquired by the optical image acquisition unit 201 as input data for a trained model. Here, the skeletal information is information indicating a plurality of predefined feature points on the subject, and includes, for example, coordinates indicating body parts such as the face, shoulders, hands, waist, and feet, and can be used to recognize the posture of the human body. In this embodiment, the skeleton estimation unit 202 uses a trained model constituted by a machine learning model typified by a neural network. Here, the trained model is a model generated in advance by supervised machine learning using a variety of image data not limited to the radiography site, and is additionally trained with data corresponding to the output desired in the radiography site.

[0047] First, in the case of the head, the feature points can include at least one of the eyes, nose, and mouth, for example, to distinguish between the front and back of the head. Furthermore, in the case of the torso, the feature points can include, for example, feature points for distinguishing between the neck, chest (thoracic vertebrae), abdomen, and lumbar region (lumbar vertebrae). Furthermore, in the case of the limbs, the feature points can include, for example, shoulders, elbows, wrists, hip joints, knees, and ankles. The skeleton estimation unit 202 estimates skeleton information indicating these feature points, allowing the information processing device 200 to recognize a wide range of postures. Furthermore, parts with laterality, such as the right hand and the left hand, can be defined as different feature points at the time of skeleton estimation. In this case, the information processing device 200 can estimate laterality based on the feature points.

[0048] The skeleton estimation process may be performed on the optical image using a single neural network model, or may include a subject region extraction process for extracting a subject region from the optical image and performing skeleton estimation from the extracted subject region. In this case, the subject region in the optical image may be used as input data for the trained model. Alternatively, the subject region in the optical image may be used as training data. Furthermore, the skeleton estimation process may include a classification process for classifying whether the optical image shows the subject's entire body or only parts such as the head, hands, or feet. In this case, the skeleton estimation unit 202 may select and apply a trained model for skeleton estimation specialized for each class, such as the whole body, head, arms, or feet, based on the results of the classification process.

[0049] Typically, neural network models used for skeletal estimation are trained models for general-purpose use, which are machine-learned using large amounts of data including optical images and skeletal information of the subject in the optical images. However, when such trained models are used as skeletal estimation units required in the medical settings of this embodiment, there is a possibility that the number of feature points that can be output may be insufficient. For example, it is possible that the trained model is trained to output the shoulder joints, elbow joints, wrists, hip joints, knee joints, and ankles of the limbs, but is unable to output feature points representing the fingers, toes, and even the front and back of the fingers and toes. Therefore, in this embodiment, a trained model that has been additionally trained on such a general-purpose trained model is used to output feature points required for each medical institution or medical setting. Additional training of the neural network model to output additional feature points for certain human body parts can be achieved by machine learning using a dataset that is relatively small compared to the dataset used to train the general-purpose trained model. For example, additional training can be performed using a dataset necessary and sufficient to output the feature points required for each medical institution or medical setting where the radiography system 100 is used.

[0050] The training data used for additional training may include a dataset in which an optical image is used as input data and, for feature points desired for additional training, labels indicating the feature points are attached to the positions of the feature points in the optical image, or an image is used as output data (correct answer data). Note that the correct answer data may be generated by a doctor or the like based on the optical image. When an optical image is input, a trained model trained using such training data can output the coordinates of each feature point in the input optical image, including the additionally trained feature points, and a probability indicating the likelihood (likelihood) of the feature point. Note that if the probability is 0.0, the trained model does not need to output the position of the feature point. Furthermore, training data of a general-purpose trained model before additional training may also include datasets in a similar format, but the training data used for additional training includes skeletal information regarding a skeleton different from the skeleton indicated by the skeletal information trained by the general-purpose trained model. Note that the trained model used by the skeleton estimation unit 202 may have already undergone additional training, and training does not need to be performed each time processing is performed.

[0051] In this embodiment, an example will be described in which the skeleton estimation unit 202 applies arm skeleton estimation, which estimates the left wrist, left elbow joint, left shoulder joint, right wrist, right elbow joint, and right shoulder joint as skeleton information, to the optical image 400. In this example, the skeleton estimation unit 202 inputs the optical image 400 to the trained model. In this case, as shown in FIG. 5, the trained model detects the left wrist 501 and the right wrist 502 and outputs their respective coordinates (x501, y501) and (x502, y502) and probabilities P501 and P502 representing their likelihood as feature points. In contrast, the trained model estimates that the probability that the left shoulder joint, left elbow joint, and right shoulder joint are included in the optical image 400 is 0.0 and does not output the corresponding coordinates. On the other hand, the trained model estimates that the right elbow joint 503 is present at the coordinates (x503, y503) with a probability of P503. Here, it is assumed that the estimation regarding the right elbow joint is incorrect, and in this case, probability P503 will be smaller than probabilities P501 and P502.

[0052] The skeleton estimation unit 202 outputs the coordinates and probability of each feature point as skeleton information in a format like table 500. Note that the skeleton estimation unit 202 may output the skeleton information in a format according to the output of the trained model. For example, the trained model may output matrix data in a format like table 500, and the skeleton estimation unit 202 may output the matrix data or may output it in a format like table 500. Furthermore, for example, the trained model may output a map (heat map) that visualizes feature amounts extracted from an optical image. In this case, the skeleton estimation unit 202 may output the map, or may output the position and probability of the most probable feature point in the map in a format like table 500.

[0053] (Step S303) In step S303, the subject information determination unit 203 determines the laterality of the subject, who is the subject of radiographic imaging, from the skeletal information estimated by the skeleton estimation unit 202, and outputs the result as subject information. Specifically, the subject information determination unit 203 applies a rule-based algorithm such as that shown in Fig. 6 to the skeletal information estimated by the skeleton estimation unit 202. In this rule-based algorithm, the laterality of the subject is determined based on the probability and coordinates of the skeletal information output by the skeleton estimation unit 202.

[0054] Here, an example will be described in which the rule-based algorithm of Fig. 6 is applied to the output of the skeleton estimation shown in Fig. 5. The output of the skeleton estimation shown in Fig. 5 includes a left wrist 501, a right wrist 502, and a right elbow joint 503, and the probability P503 of the right elbow joint 503 is assumed to be small.

[0055] First, in step S601, the object information determination unit 203 deletes feature points whose probability of being a feature point is less than a threshold value from among the feature points included in the skeleton information output by the skeleton estimation unit 202. In the above example, the object information determination unit 203 determines that the probability P503 of the right elbow joint 503 is less than the threshold value and deletes the right elbow joint 503 as a feature point.

[0056] Next, in step S602, the object information determination unit 203 determines whether the number of remaining feature points is one or more. If it is determined in step S602 that the number of feature points is more than one, the process proceeds to step S603. On the other hand, if it is determined in step S602 that the number of feature points is one, the process proceeds to step S604. In the above example, the object information determination unit 203 determines that the number of remaining feature points is two, that is, the left wrist 501 and the right wrist 502, and therefore the process proceeds to step S603.

[0057] In step S603, the object information determination unit 203 determines the position of the planned radiation irradiation area (irradiation area) in the optical image, and deletes the remaining feature points except for the feature point closest to the position of the planned radiation irradiation area. In the above example, the object information determination unit 203 calculates the distance between the planned radiation irradiation area and the coordinates (x501, y501) of the left wrist, and the distance between the planned radiation irradiation area and the coordinates (x502, y502) of the right wrist.

[0058] Here, the position of the region to be irradiated with radiation in the optical image 400 may be the position of the collimator lamp irradiation region 404. Therefore, the object information determination unit 203 may extract the collimator lamp irradiation region 404 by, for example, threshold processing based on the brightness of the optical image, and calculate the distance from the feature point by regarding the collimator lamp irradiation region 404 as the region to be irradiated with radiation. Note that the method for extracting the collimator lamp irradiation region 404 in the optical image is not limited to this, and any known method may be used. For example, the object information determination unit 203 may extract the collimator lamp irradiation region 404 by using edge detection, corner detection, or the like.

[0059] The distance dn between the coordinates (xn, yn) of the feature point n and the representative point (for example, the center position (xc, yc)) of the region to be irradiated with radiation can be calculated according to the following formula 1.

number

[0060] The subject information determination unit 203 compares the distance between each feature point and the planned radiation irradiation area, calculated using Equation 1, to find the feature point closest to the planned radiation irradiation area, and can delete feature points other than this feature point. In the above example, the right wrist 502 is closest to the planned radiation irradiation area among the skeleton estimation outputs shown in Figure 5, so the subject information determination unit 203 deletes the left wrist 501, which is a feature point.

[0061] In step S604, the subject information determination unit 203 outputs the laterality of the remaining feature points as subject information. In the above example, the subject information determination unit 203 outputs the laterality "right" of the right wrist 502, which is the remaining feature point. When the laterality of the feature points is output in step S604, the laterality determination process, which is the subject information determination process according to this embodiment, ends.

[0062] In the subject information determination process, the correspondence between the remaining feature points and the feature points to be output may be determined in advance by a rule. For example, if the right elbow joint or the right wrist remains as a feature point, the output laterality may be set to "right," and if the left elbow joint or the left wrist remains as a feature point, the output laterality may be set to "left."

[0063] Although the above description has been given of the case where the feature points are the right wrist and the left wrist, other feature points such as shoulder joints, elbow joints, hip joints, knee joints, or ankles may also be used. In this case, too, the object information determination unit 203 can output, as object information, the laterality of the feature points included in the skeleton information output by the skeleton estimation unit 202.

[0064] Furthermore, the rules for the process of determining object information are not limited to the above rules. For example, in the process of determining object information, if the remaining feature point is a "right shoulder joint" or a "left shoulder joint," and the laterality to be determined is a laterality related to a "hand," some kind of error may be considered to have occurred, and therefore a rule may be considered in which an error is output.

[0065] Furthermore, the planned radiation irradiation region is not limited to the collimator lamp irradiation region, and may be determined using other known methods. For example, more simply, the center of the optical image may be determined as the planned radiation irradiation region. In this case, the subject information determination unit 203 may output the laterality of a feature point close to the center of the optical image as the subject information. In this case, the laterality of the subject can be determined even if the collimator lamp irradiation region is not captured in the optical image. Furthermore, for example, the planned radiation irradiation region may be determined by providing a marker or the like on the imaging table to indicate the planned radiation irradiation region, and extracting the marker from the optical image using the subject information determination unit 203.

[0066] Furthermore, if higher accuracy is required in determining the object information, the radiation irradiation target region in the image may be calculated taking into account the spatial positions of the radiation generation device 120 and the camera 140. For example, as shown in Fig. 7, consider a camera coordinate system in which the optical center of the camera 140 is the origin (0,0,0), the optical axis direction of the camera 140 is the Z axis direction, and the horizontal and vertical directions of the image are the X axis direction and the Y axis direction, respectively. Here, consider a case in which the spatial coordinates (X120, Y120, Z120) of the radiation generation device 120 and the radiation irradiation direction (vx, vy, vz) are known.

[0067] This information can be obtained, for example, by installing the camera 140 relative to the radiation generation device 120 while measuring the position with a tape measure or the like. Alternatively, the information may be obtained based on the amount of drive from the installation position of the radiation generation device 120 using a configuration capable of mechanically determining the amount of drive, such as a stepping motor. Furthermore, the information may be obtained by attaching a gyro mechanism, an acceleration sensor, or the like to the radiation generation device 120 to acquire the amount of displacement of the position or angle from the initial position and inputting the acquired information to the information processing device 200. However, when the camera 140 is installed relative to the radiation generation device 120, the spatial coordinates (X120, Y120, Z120) and the radiation irradiation direction (vx, vy, vz) of the radiation generation device 120 in the camera coordinate system after installation can generally be considered to be fixed values. In this case, the coordinates (xp, yp) of the area to be irradiated with radiation in the acquired optical image can be expressed by the following equation 2, where D is the distance between the radiation generating device 120 and the radiation detector 130, (fx, fy) is the focal length of the camera 140, and (cx, cy) is the optical center.

number

[0068] By determining the planned radiation irradiation region (xp, yp) as described above, the subject information determination unit 203 can more accurately determine the laterality of the subject by outputting the laterality of the feature point closest to the planned radiation irradiation region (xp, yp) as subject information.

[0069] (Example of use of determined specimen information) An example of processing using the subject information obtained above will be described with reference to Fig. 8. In this example, when processing using the subject information is started, the subject information obtained in the subject information determination processing is transferred to the consistency determination unit 204 via, for example, the CPU bus 230, and the processing proceeds to step S801. In step S801, the consistency determination unit 204 determines the consistency between the imaging order information and the subject information. For example, when the imaging target part of the imaging order is "hand" and the laterality is "right," the consistency determination unit 204 outputs "consistency" if the laterality included in the obtained subject information is "right," and outputs "inconsistency" otherwise.

[0070] In step S802, the display control unit 207 causes the display unit 235 to display the optical image acquired by the optical image acquisition unit 201, the object information output by the object information determination unit 203, the imaging order, and the consistency determination result. The display control unit 207 can also cause the display unit 235 to display the object region and skeletal information acquired by the skeleton estimation unit 202, and analysis information obtained during the object information determination process.

[0071] In step S803, the operator checks the information displayed on the display unit 235, and then inputs a radiographic image capturing instruction to the information processing device 200 via the operation unit 234. The radiographic image acquiring unit 205 acquires a radiographic image in response to the radiographic image capturing instruction from the operator. Specifically, the radiographic image acquiring unit 205 causes the radiation generator 120 to irradiate a radiation beam under imaging conditions according to the imaging order, and acquires a radiographic image from the radiation detector 130 that detects the radiation that has transmitted through the subject O. The radiographic image acquiring unit 205 may also acquire a radiographic image in response to the completion of the consistency determination process. In this case, the radiographic image acquiring unit 205 may acquire a radiographic image corresponding to the optical image used in the subject information determination process from the external storage device 160, the storage unit 232, etc.

[0072] In step S804, the subject information is transferred to the annotation unit 206 via, for example, the CPU bus 230, and the annotation unit 206 annotates the acquired radiographic image with the subject information. For example, because radiographic images are fluoroscopic images, it is difficult to determine whether a radiographic image of a right hand is a right hand or a left hand. However, by the annotation unit 206 annotating the radiographic image with information such as "right" as laterality information, the radiographic image can be provided with information that the subject depicted in the radiographic image has a right hand. The radiographic image annotated with the subject information can be transferred to the display control unit 207 or the storage unit 232 via the CPU bus 230. The annotation unit 206 may also annotate the subject information on the optical image. In this case, the optical image annotated with the subject information can be transferred to the display control unit 207 or the storage unit 232 via the CPU bus 230.

[0073] In step S805, the display control unit 207 can display the radiographic image annotated with the subject information on the display unit 235. The display control unit 207 can display the optical image and the radiographic image annotated with the subject information side by side or by switching between them on the display unit 235. When the display process of the radiographic image is completed, the process using the subject information is completed. The display unit 235 can also display the optical image annotated with the subject information as the optical image.

[0074] As described above, the radiation imaging system 100 according to this embodiment includes a radiation generator 120 and a radiation detector 130 that perform radiation imaging of a subject, a camera 140 that functions as an example of an optical device that captures an optical image of the subject, and an information processing device 200. The information processing device 200 includes an optical image acquisition unit 201, a skeleton estimation unit 202, and a subject information determination unit 203. The optical image acquisition unit 201 functions as an example of an acquisition unit that acquires an optical image obtained by imaging the subject at a radiation image imaging site. The skeleton estimation unit 202 functions as an example of an estimation unit that estimates skeletal information of the subject in the acquired optical image by using the acquired optical image as input data for a second trained model that is obtained by additionally training a general-purpose trained model (first trained model) that estimates skeletal information about the subject's skeleton using skeletal information about a skeleton different from the skeletal information learned by the first trained model. The object information determination unit 203 functions as an example of a determination unit that determines object information including information on the laterality of the object in the optical image, using skeletal information of the object in the optical image.

[0075] With the above configuration, the information processing device 200 according to this embodiment can determine the laterality of the subject (radiography subject) as subject information using skeletal information estimated from an optical image. Therefore, the information processing device 200 can more appropriately assist the operator in determining whether the subject's laterality is appropriate using the determined subject information. Furthermore, the determined subject information can be used as additional information provided to the operator along with live images during radiography for medical examinations. Furthermore, the trained model used to estimate skeletal information according to this embodiment is obtained by additionally training a general-purpose trained model according to the purpose of each medical institution or medical site. Therefore, the information processing device 200 can appropriately estimate skeletal information required by medical institutions or medical sites without collecting a large amount of training data.

[0076] The subject information determination unit 203 can determine the subject information by rule-based processing. Therefore, the information processing device 200 can determine the subject information by rule-based processing, which is relatively easy to adjust, without adjusting a trained model, the configuration of which is difficult to change. Therefore, the information processing device 200 can determine the subject information by rules according to the operation and knowledge of a medical institution or medical site.

[0077] Furthermore, the skeleton estimation unit 202 can estimate, as skeleton information of the subject in the optical image, the coordinates of a plurality of feature points of the subject in the optical image and the probability that the plurality of feature points correspond to predefined feature points of the subject. In this case, the subject information determination unit 203 can select feature points of the subject in the optical image using a threshold value for the probability estimated by the skeleton estimation unit 202, and determine subject information using the selected feature points. By selecting feature points in this way, the information processing device 200 can determine subject information based on more appropriate feature points.

[0078] Furthermore, the object information determination unit 203 can determine a radiation irradiation target region (irradiation region) to be irradiated with radiation in the optical image. The object information determination unit 203 can select feature points of the object in the optical image based on the positional relationship between the coordinates estimated by the skeleton estimation unit 202 and the irradiation region, and determine object information using the selected feature points. By selecting feature points in this way, the information processing device 200 can determine object information based on more appropriate feature points.

[0079] The object information determination unit 203 can determine the irradiation area based on the irradiation area of ​​the collimator lamp in the optical image or the center of the optical image. The object information determination unit 203 can also determine the irradiation area based on the arrangement of the radiation generation device 120 that irradiates radiation and the camera 140 that functions as an example of an optical device that generates an optical image. In this case, by taking into account the arrangement of the radiation generation device 120 in the camera coordinate system, appropriate analysis processing can be performed according to the position of the device in each medical setting, and the analysis accuracy of the object information determination unit 203 can be improved.

[0080] The skeleton estimation unit 202 can also extract a subject region in the optical image and estimate skeletal information of the subject in the optical image by using the subject region in the optical image as input data for the second trained model. The skeleton estimation unit 202 can also classify the optical image into classes indicating whether the image shows the subject's entire body, head, hands, or feet, and select and use a second trained model corresponding to the class. In these cases, the skeleton estimation unit 202 can be expected to estimate more appropriate skeletal information.

[0081] The second trained model may be a trained model obtained by additional training using a smaller number of training data than the number of training data of the first trained model. For example, the second trained model may be a trained model obtained by additional training using skeletal information on at least one feature point of the left and right elbow joints, wrists, fingers, the front and back of the hands, knee joints, ankles, toes, and the front and back of the feet.

[0082] The information processing device 200 may further include a consistency determination unit 204 that functions as an example of a determination unit that determines whether or not there is consistency between the subject information and information included in the radiographic image capturing order. In this case, the information processing device 200 can more appropriately support the operator in determining whether or not the imaging conditions, such as the subject's posture, correspond to the radiographic image capturing order by presenting the determination result by the consistency determination unit 204.

[0083] In this regard, the information processing device 200 may further include a display control unit 207 that causes the display unit 235 to display at least one of the optical image, the subject skeletal information in the optical image, the subject region in the optical image, the subject information, and the determination result by the consistency determination unit 204. Note that, when the consistency determination unit 204 outputs a determination result indicating inconsistency, the display control unit 207 can cause the display unit 235 to display a warning. In this case, the information processing device 200 can notify the operator that inconsistency is not achieved and prompt the operator to adjust the imaging conditions, such as the subject's posture, in accordance with the imaging order, thereby more appropriately supporting radiography.

[0084] The information processing device 200 may further include an annotation unit 206 that arranges subject information on an optical image or a radiological image as annotation information. In this case, the information processing device 200 can more appropriately support the operator in determining the imaging conditions, such as the subject's posture, by presenting the optical image or the radiological image on which the annotation information is arranged. The display control unit 207 can display the annotation information and the optical image or the radiological image on which the annotation information is arranged on the display unit 235.

[0085] Example 2 Hereinafter, a radiography system, an information processing device, and an information processing method according to a second embodiment of the present disclosure will be described with reference to FIGS. 9 to 14. In this embodiment, a process of recognizing a region of a subject to be imaged from optical images acquired at a predetermined frame rate using a video camera as the camera 140 and outputting the region as subject information will be described. Here, the region of the subject includes, for example, the head, chest, and limbs of a human body. Furthermore, the granularity of the region to be recognized may vary depending on the purpose, such that, for example, the limbs can be divided into smaller regions such as the upper arm, elbow, wrist, hand, and finger. In particular, a case of determining whether the region to be imaged in an imaging order is the "chest" or the "abdomen" will be described as an example.

[0086] The configurations of the radiation imaging system and the information processing device according to this embodiment are the same as those of the radiation imaging system and the information processing device according to the first embodiment, and therefore the same reference numerals are used and the description thereof is omitted. Also, the description of the same processing as that described in detail in the first embodiment is omitted.

[0087] (Processing flow) A series of processing steps according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the processing steps according to this embodiment. When the processing steps according to this embodiment start, the process proceeds to step S901.

[0088] (Step S901) In step S901, the optical image acquisition unit 201 controls the camera 140 to acquire optical images of the radiation imaging site, including the subject being radiographed. In this embodiment, the camera 140 is a video camera attached to a radiation generator, and captures an image of the subject taking an imaging position on the radiation detector 130 arranged on a supine table, and outputs optical images at a predetermined frame rate.

[0089] Here, an example will be described with reference to Fig. 10 in which a radiation detector 130, a head 1002, a chest 1003, and an abdomen 1004 are captured in an optical image 1000. Fig. 10 shows an example of an optical image according to this embodiment. Here, a radiation detector region 1001 in the optical image 1000 is a region representing the radiation detector 130. Furthermore, in the example shown in Fig. 10, it is assumed that light from a collimator lamp is irradiated onto the chest of the subject, and a collimator lamp irradiation region 1005 is depicted above the chest 1003 in the optical image 1000. The collimator lamp irradiation region 1005 coincides with the region to be irradiated with radiation during radiography.

[0090] (Step S902) In step S902, the skeleton estimation unit 202 estimates skeletal information of the subject by using the optical image acquired by the optical image acquisition unit 201 as input data for the trained model. In this embodiment, an example will be described in which the skeleton estimation unit 202 applies whole-body skeleton estimation, in which the skeleton estimation unit 202 estimates left and right eyes, left and right shoulder joints, left and right hip joints, left and right elbows, left and right wrists, left and right knees, and left and right ankles as skeletal information, to the optical image 1000. As shown in Fig. 11 , the skeleton estimation unit 202 detects left and right eyes 1101, 1102, left and right shoulder joints 1103, 1104, and left and right hip joints 1105, 1106 in the optical image 1000 as feature points.

[0091] The skeleton estimation unit 202 outputs the coordinates and probabilities of each feature point as skeleton information in a format such as Table 1100. Here, it is assumed that the skeleton estimation unit 202 outputs the coordinates (x1101, y1101) to (x1106, y1106) of each feature point and the probabilities P1101 to P1106 representing the likelihood of each feature point. However, it is assumed that the probability that other skeleton information is included in the optical image 1000 is 0.0, and the coordinates corresponding to each feature point are not output. Also, for simplicity of explanation, unlike the example of the first embodiment, it is assumed that no erroneous estimation is made here. Note that the skeleton estimation unit 202 may output skeleton information in a format corresponding to the output of the trained model, similar to the skeleton estimation unit 202 according to the first embodiment.

[0092] Furthermore, the trained model used in this example may be a trained model that has undergone additional training to a trained model used for general-purpose skeletal estimation so as to be able to output feature points required for each medical institution or medical site, similar to the trained model according to Example 1. Furthermore, the training data may also be prepared similarly to the training data according to Example 1.

[0093] (Step S903) In step S903, the subject information determination unit 203 determines the part of the subject that is the subject of radiographic imaging from the skeletal information estimated by the skeleton estimation unit 202, and outputs the determined part as subject information. Specifically, the subject information determination unit 203 applies a rule-based algorithm as shown in Fig. 12 to the skeletal information estimated by the skeleton estimation unit 202. In this rule-based algorithm, the part of the subject is determined based on the probability and coordinates of the skeletal information that is output from the skeleton estimation unit 202.

[0094] Here, an example will be described in which the rule-based algorithm of Fig. 12 is applied to the output of the skeleton estimation shown in Fig. 11 to determine whether the subject's body part is the chest or abdomen. The output of the skeleton estimation shown in Fig. 11 includes left and right eyes 1101, 1102, left and right shoulder joints 1103, 1104, and left and right hip joints 1105, 1106, and each of the probabilities P1101 to P1106 is assumed to be equal to or greater than a predetermined threshold.

[0095] First, in step S1201, the object information determination unit 203 deletes feature points whose probability of being a feature point is less than a threshold value from among the feature points included in the skeleton information output by the skeleton estimation unit 202. In the above example, since all of the probabilities P1101 to P1106 are equal to or greater than a predetermined threshold value, there are no feature points that are deleted as feature points below the threshold value.

[0096] Next, in step S1202, the object information determination unit 203 determines whether the number of remaining feature points is one or more. If it is determined in step S1202 that the number of feature points is more than one, the process proceeds to step S1203. On the other hand, if it is determined in step S1202 that the number of feature points is one, the process proceeds to step S1204. In the above example, the object information determination unit 203 determines that the number of remaining feature points is more than one, and therefore the process proceeds to step S1203.

[0097] In step S1203, the object information determination unit 203 obtains the position of the region to be irradiated with radiation in the optical image, and deletes, from the remaining feature points, those feature points other than the feature point closest to the position of the region to be irradiated with radiation. In the above example, the object information determination unit 203 calculates the distance between the position of the region to be irradiated with radiation and the coordinates (x1101, y1101) to (x1106, y1106) of the feature points.

[0098] Here, the position of the region to be irradiated with radiation in the optical image 1000 may be the collimator lamp irradiation region 1005. Therefore, similar to the first embodiment, the object information determination unit 203 may extract the collimator lamp irradiation region 1005 by threshold processing based on the brightness or the like of the optical image, and calculate the distance from the feature point by regarding the collimator lamp irradiation region 1005 as the region to be irradiated with radiation. Note that the region to be irradiated with radiation may be calculated taking into consideration the center of the optical image and the spatial positions of the radiation generation device 120 and the camera 140, as described in the first embodiment.

[0099] The subject information determination unit 203 compares the distance between each feature point and the region to be irradiated with radiation, finds the feature point that is closest to the region to which radiation sickness is present, and can delete feature points other than that feature point. In the above example, of the skeleton estimation outputs shown in Figure 11, either the left or right shoulder joint 1103 or 1104 is closest to the region to be irradiated with radiation, so the subject information determination unit 203 deletes the other feature points.

[0100] In step S1204, the object information determination unit 203 outputs the location of the remaining feature points as object information. In the above example, the object information determination unit 203 outputs the location corresponding to one of the left and right shoulder joints 1103, 1104, which is the remaining feature point. Since the rule-based algorithm according to this embodiment is an algorithm for determining whether the location is the chest or the abdomen, the determination result output is "chest," which is close to the shoulder joint. In the algorithm for determining whether the location is the chest or the abdomen, "chest" is also output even if the feature point closest to the planned radiation irradiation region is either the left or the right eye. On the other hand, "abdomen" is output when the feature point closest to the planned radiation irradiation region is the left or right hip joints 1105, 1106, or, although not shown here, the left or right wrist or knee joint. When the location of the object is output in step S1204, the laterality determination process, which is the process for determining object information according to this embodiment, is completed.

[0101] The above describes a case where feature points are the left and right eyes, shoulder joints, and hip joints, and the subject's region is determined to be the chest or abdomen. However, the subject information determination unit 203, which outputs the region as subject information, may use other feature points or determine other regions, such as the head or limbs. The rule-based algorithm for determining subject information may vary depending on the imaging posture adopted by the medical institution or medical site, the feature points captured within the angle of view of the camera 140 used, and other factors. For example, a rule-based algorithm intended to determine the head and limbs may output "head" if the feature point closest to the radiation irradiation target region is either the left or right eye. Furthermore, a rule-based algorithm intended to determine the left and right wrists and knee joints may output regions with granularity such as "left wrist," "right wrist," "left knee joint," and "right knee joint."

[0102] Other rule-based algorithms for determining the body part of a subject are also possible. For example, as shown in Fig. 13, it is assumed that a certain medical institution installs the camera 140 so that the optical image obtained when imaging the chest is optical image 1300 and the optical image obtained when imaging the abdomen is optical image 1301. In this case, the subject information determination unit 203 may apply a rule-based algorithm such as that shown in Fig. 14 to the output of the bone structure estimation.

[0103] In the rule-based algorithm shown in FIG. 14, first, in step S1401, the object information determination unit 203 simply determines whether there are any feature points belonging to the head. If it is determined in step S1401 that there are any feature points belonging to the head, the process proceeds to step S1402. In step S1402, the object information determination unit 203 outputs "chest" as the object information. On the other hand, if it is determined in step S1401 that there are no feature points belonging to the head, the process proceeds to step S1403. In step S1403, the object information determination unit 203 outputs "abdomen" as the object information. When the part of the object is output in step S1402 or step S1403, the laterality determination process, which is the object information determination process according to this embodiment, ends.

[0104] In this way, the rule-based algorithm for determining subject information may be adapted to the operations at the medical institution or medical site where the radiation imaging system 100 or the information processing device 200 is used. Here, the operations at the medical institution or medical site may include, for example, agreements regarding the posture of the subject during radiation imaging and the placement of the camera 140 depending on the imaging region.

[0105] (Example of use of determined specimen information) The subject information including the subject's body part obtained as described above may be used for processing such as annotation, as in the first embodiment. For example, the subject information including the subject's body part obtained may be transferred to the consistency determination unit 204 via the CPU bus 230 and used to determine the consistency between the imaging order information and the subject information. The display control unit 207 can also display the subject information and analysis information obtained along the way on the display unit 235. Furthermore, the annotation unit 206 can annotate the subject information on the acquired radiographic image. The display control unit 207 can also display the radiographic image annotated with the subject information on the display unit 235.

[0106] As described above, the subject information determination unit 203 can determine subject information including information indicating the region of the subject in the optical image, using skeletal information of the subject in the optical image. With the above configuration, the information processing device 200 according to this embodiment can determine the region of the subject (radiography subject) as subject information, using skeletal information estimated from the optical image. Therefore, the information processing device 200 can more appropriately support the operator's judgment regarding the appropriateness of the region of the subject, using the determined subject information. Furthermore, as in the first embodiment, the determined subject information can be used as additional information to be provided to the operator together with a live image in radiography for medical examinations. Furthermore, as in the first embodiment, the information processing device 200 can appropriately estimate skeletal information required in medical institutions and medical settings without collecting a large amount of training data. In this embodiment, the subject information including information indicating the region of the subject is determined. However, information indicating the laterality and region of the subject may also be determined as subject information.

[0107] Example 3 A radiation imaging system, an information processing device, and an information processing method according to a third embodiment of the present disclosure will be described below with reference to Fig. 15. In this embodiment, a customization (adjustment) process will be described for recognizing the laterality and part of the subject to be imaged from optical images acquired at a predetermined frame rate using a video camera as the camera 140, and outputting the results as subject information.

[0108] The configurations of the radiation imaging system and the information processing device according to this embodiment are the same as those of the radiation imaging system and the information processing device according to the first embodiment, and therefore the same reference numerals are used and the description thereof is omitted. Also, the description of the same processing as that described in detail in the first and second embodiments is omitted.

[0109] (Processing flow) The processing procedure of the adjustment processing of the rule-based algorithm related to the object information determination processing according to this embodiment will be described with reference to Fig. 15. Fig. 15 is a flowchart showing the processing procedure according to this embodiment. When the adjustment processing of the rule-based algorithm related to the object information determination processing according to this embodiment is started, the processing proceeds to step S1501.

[0110] (Step S1501) In step S1501, the optical image acquisition unit 201 controls the camera 140 to acquire an optical image of the radiation imaging site, including the subject being radiographed. In this embodiment, the camera 140 is a video camera attached to a radiation generator, and captures an image of the subject taking an imaging posture on the radiation detector 130 placed on a supine table, and outputs optical images at a predetermined frame rate. Here, as in the second embodiment, a case will be described with reference to FIG. 10 where the radiation detector 130, head 1002, chest 1003, and abdomen 1004 are captured in the optical image 1000.

[0111] (Step S1502) In step S1502, the skeleton estimation unit 202 estimates skeletal information of the subject by using the optical image acquired by the optical image acquisition unit 201 as input data for the trained model. In this embodiment, an example will be described in which the skeleton estimation unit 202 applies whole-body skeleton estimation, in which the skeleton estimation unit 202 estimates left and right eyes, left and right shoulder joints, left and right hip joints, left and right elbows, left and right wrists, left and right knees, and left and right ankles as skeletal information, to the optical image 1000. As in the second embodiment, the skeleton estimation unit 202 can detect left and right eyes 1101 and 1102, left and right shoulder joints 1103 and 1104, and left and right hip joints 1105 and 1106 in the optical image 1000 as feature points, as shown in FIG.

[0112] Note that the trained model used in this example may be a trained model that has undergone additional training to a trained model used for general-purpose skeletal estimation so as to be able to output feature points required for each medical institution or medical site, similar to the trained model according to Example 1. Furthermore, the training data may also be prepared in the same manner as the training data according to Example 1.

[0113] (Step S1503) In step S1503, the subject information determination unit 203 adjusts parameters of a rule-based algorithm for determining a region of the subject that is the subject of radiographic imaging, in response to an input from the operator via the operation unit 234. More specifically, the display control unit 207 causes the display unit 235 to display the skeletal information estimated by the skeleton estimation unit 202. Based on the skeletal information displayed on the display unit 235, the operator instructs the information processing device 200 via the operation unit 234 which region of the subject should be output as subject information. The subject information determination unit 203 adjusts various parameters of the rule-based algorithm as shown in FIG. 12 in response to an instruction from the operator.

[0114] Here, the parameters of the rule-based algorithm may include, for example, a threshold value for threshold processing applied to the probability in step S1201, and the number of feature points serving as a criterion for determining whether to branch the process in step S1202. Furthermore, the parameters of the rule-based algorithm may include, in step S1203, the positional relationship between the planned radiation irradiation region and the feature points to be retained, and the planned radiation irradiation region, and the correspondence relationship between the remaining feature points and the region to be output in step S1204. Note that the positional relationship between the feature points to be retained and the planned radiation irradiation region may include, for example, the order of proximity of the feature points to be retained to the planned radiation irradiation region (e.g., closest or second-closest).

[0115] For example, a case will be described in which, in order to adjust the rule-based algorithm as shown in FIG. 12 , the operator inputs an instruction via the operation unit 234 indicating that the region to be determined from the estimated skeletal information is the "chest." In this case, the object information determination unit 203 determines the planned radiation irradiation region and determines which of the left and right eyes 1101 and 1102, the left and right shoulder joints 1103 and 1104, and the left and right hip joints 1105 and 1106 is the closest feature point to the planned radiation irradiation region. In the above example, either the shoulder joints 1103 or 1104 is closest. Therefore, the object information determination unit 203 adjusts the parameters of the rule-based algorithm so that, when the shoulder joints are closest to the planned radiation irradiation region, the object information determination unit 203 determines and outputs the object information as the "chest." In this case, the object information determination unit 203 can adjust the positional relationship between the feature points to be retained and the planned radiation irradiation region, for example, in step S1203, so as to determine either the shoulder joints 1103 or 1104 as the feature point to be retained. Furthermore, the object information determination unit 203 can adjust the correspondence between the remaining feature points in step S1204 and the region names to be output.

[0116] Alternatively, subject information determination unit 203 may store, in storage unit 232, the coordinates of feature points with a probability equal to or greater than a threshold, among the skeletal information output by skeleton estimation unit 202, together with the region information input by the operator, without calculating the planned radiation irradiation region. In this case, information such as the coordinates of feature points when the region information is registered as the chest, and the coordinates of feature points when the region information is registered as the abdomen, is stored in storage unit 232. In such a case, when inputting an optical image to determine the region of the subject, subject information determination unit 203 may calculate the sum of the distances between the coordinates of feature points obtained by inputting the optical image and the coordinates of the stored feature points, and determine the region with the smallest sum as subject information.

[0117] Regarding a method for adjusting parameters of the rule-based algorithm, a configuration may be adopted in which multiple parameter combinations are prepared in advance and one of the parameter combinations is selected in response to an instruction from an operator. For example, the object information determination unit 203 may select a parameter combination by referring to a prepared lookup table in response to an instruction from an operator, and adjust the parameters. The parameter combinations may include, for example, a parameter combination related to processing as shown in FIG. 12, a parameter combination related to processing using coordinates of feature points equal to or greater than a threshold, and a parameter combination related to processing as shown in FIG. 14. The parameter combinations may also include, for example, a parameter combination related to processing for determining the laterality of the object as shown in FIG. 12. The instruction from the operator may specify object information to be output as described above, or may specify a parameter combination.

[0118] In step S1503, the parameters of the rule-based algorithm are adjusted, and the series of processes ends.

[0119] In this embodiment, an example has been described in which parameters of a rule-based algorithm are adjusted for the process of determining the region of a subject as subject information. In contrast, for the process of determining the laterality of a subject as subject information, such as in Example 1, parameters of the rule-based algorithm can be adjusted in the same manner as in this embodiment. In this case, the parameters of the rule-based algorithm may include, for example, a threshold value for the threshold process applied to the probability in step S601 and the number of feature points serving as a criterion for determining whether to branch the process in step S602. Furthermore, the parameters of the rule-based algorithm may include the positional relationship between the region to be irradiated and the feature points to be retained in step S603 and the region to be irradiated, and the correspondence relationship between the remaining feature points and the laterality to be output in step S604. For the process of determining the region and laterality of a subject as subject information, parameters of the rule-based algorithm can also be adjusted in the same manner as in this embodiment.

[0120] As described above, the object information determination unit 203 according to this embodiment can adjust parameters of the rule-based processing based on skeletal information of the object in the optical image and object information acquired via the operation unit 234. Here, the parameters of the rule-based processing can include at least one of a threshold value for threshold processing related to selecting feature points of the object in the optical image, a positional relationship between the feature points and an irradiation area to which radiation is irradiated, and a correspondence relationship between the feature points and at least one of a region and a side of the object. With the above configuration, the information processing device 200 according to this embodiment can flexibly adapt the object information determination process to the operations and knowledge of the medical institution or medical site by adjusting the rules according to the operations and knowledge of the medical institution or medical site.

[0121] In this embodiment, as in Embodiments 1 and 2, the optical image acquisition unit 201 has been described as acquiring an optical image of a radiation imaging site. Alternatively, the optical image used in the adjustment process of the rule-based algorithm may be an optical image acquired at a virtual radiation imaging site by imaging a human body phantom. That is, the subject information determination unit 203 can adjust the parameters of the rule-based processing based on skeletal information acquired using the optical image acquired by imaging the phantom and subject information acquired via the operation unit 234. In this case, an image that makes it easier to estimate the skeleton in step S1502 can be acquired, compared to using an actual subject, and more appropriate parameter adjustment can be performed in step S1503.

[0122] Furthermore, the optical image used in the adjustment process of the rule-based algorithm does not have to be an actual image. For example, a two-dimensional image on which three-dimensional coordinates representing virtual skeletal positions calculated from virtual subject data generated using a three-dimensional modeling tool are projected may be used. In this case, the subject information determination unit 203 may use skeletal information output by the skeleton estimation unit 202 based on the two-dimensional image for the parameter adjustment process. That is, the subject information determination unit 203 can adjust the parameters of the rule-based processing based on skeletal information obtained by projecting the three-dimensional coordinates of virtual subject data generated by the three-dimensional modeling tool onto two-dimensional image coordinates and subject information acquired via the operation unit 234. This method allows an image showing skeletal positions in a desired pose to be relatively easily acquired on the information processing device 200 without having to position the human body or phantom in an arbitrary pose. Note that such two-dimensional images may be used as input data for training data for additional training of a trained model used by the skeleton estimation unit 202.

[0123] In this embodiment, the subject information determination unit 203 adjusts parameters of a rule-based algorithm for determining subject information in response to instructions from the operator. Alternatively, the information processing device 200 may be provided with a programming environment, allowing the operator to write a program for handling the skeletal information output by the skeleton estimation unit 202 via the operation unit 234. Such a configuration may use a programming language such as C or Python. The information processing device 200 may also be configured using a no-code development platform that allows rule-based algorithms to be built using a graphical user interface. Furthermore, in recent years, it has become possible to generate programs by inputting instructions in natural language using an artificial intelligence chatbot, a type of generative AI. Therefore, the information processing device 200 may be configured to utilize a generative AI such as an artificial intelligence chatbot.

[0124] Furthermore, the training data for the trained model is not limited to data obtained using the camera 140 itself that actually takes the images, but may be data obtained using a camera of the same model or type, depending on the desired configuration. Note that the trained model for skeletal estimation according to the above-described embodiments and examples is considered to extract, as part of the features, the magnitude of the luminance values ​​of the optical image, the order and gradient of the bright and dark areas, position, distribution, continuity, etc., and use them in the process of estimating skeletal information.

[0125] The trained model for skeleton estimation described above can be provided in the information processing device 200. The inference device (trained model) may be configured, for example, as a software module executed by a processor such as a CPU, MPU, GPU, or FPGA, or as a circuit performing a specific function such as an ASIC. The inference device may also be provided in a device such as a separate server connected to the information processing device 200. In this case, the information processing device 200 can use the inference device by connecting to a server equipped with the inference device via any network such as the Internet. Here, the server equipped with the inference device may be, for example, a cloud server, a fog server, or an edge server. Note that when a network within a facility, a site including a facility, or a region including multiple facilities is configured to enable wireless communication, the reliability of the network may be improved by using radio waves in a dedicated wavelength band allocated exclusively to the facility, site, region, or the like. The network may also be configured using wireless communication capable of high-speed communication, large-capacity communication, low-latency communication, and multiple simultaneous connections.

[0126] (Other Examples) The present disclosure can also be realized by providing a program that implements one or more functions of the above-described embodiments and examples to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions. A computer may have one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions.

[0127] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). The processor or circuitry may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0128] The above disclosure includes the following configurations, methods, and programs. (Configuration 1) an acquisition unit that acquires an optical image obtained by imaging a subject at a radiation image imaging site; an estimation unit that estimates skeletal information of the subject in the optical image by using the optical image as input data of a second trained model obtained by additionally training a first trained model that estimates skeletal information about the subject's skeleton using skeletal information about a skeleton different from the skeleton indicated by the skeletal information learned by the first trained model; a determination unit that determines object information including information on at least one of laterality and location of the object in the optical image using skeletal information of the object in the optical image; An information processing device comprising: (Configuration 2) 2. The information processing apparatus according to configuration 1, wherein the determination unit determines the object information by rule-based processing. (Configuration 3) 3. The information processing device according to configuration 1 or 2, wherein the estimation unit estimates coordinates of a plurality of feature points of the subject in the optical image and probabilities that the plurality of feature points correspond to predefined feature points of the subject, as skeletal information of the subject in the optical image. (Configuration 4) 4. The information processing device according to configuration 3, wherein the determination unit selects feature points of the object in the optical image by using a threshold value for the probability estimated by the estimation unit, and determines the object information by using the selected feature points. (Configuration 5) The determination unit determining an irradiation area in the optical image that is irradiated with radiation; selecting feature points of the object in the optical image based on the positional relationship between the coordinates estimated by the estimation unit and the irradiation area; 5. The information processing device according to configuration 3 or 4, wherein the object information is determined using the selected feature points. (Configuration 6) 6. The information processing device according to configuration 5, wherein the determination unit determines the irradiation area based on an irradiation area of ​​a collimator lamp in the optical image or a center of the optical image. (Configuration 7) 6. The information processing device according to configuration 5, wherein the determination unit determines the irradiation region based on an arrangement of a radiation generation device that irradiates the radiation and an optical device that generates the optical image. (Configuration 8) 8. The information processing device according to any one of configurations 1 to 7, wherein the estimation unit extracts an object region in the optical image, and estimates skeletal information of the object in the optical image by using the object region in the optical image as input data of the second trained model. (Configuration 9) 9. The information processing device according to any one of configurations 1 to 8, wherein the estimation unit classifies the optical image into a class indicating whether the optical image contains the subject's whole body, a head, a hand, or a foot, and selects and uses a second trained model corresponding to the class. (Configuration 10) 10. The information processing device according to any one of configurations 1 to 9, wherein the second trained model is a trained model obtained by performing additional training using a number of training data sets that is smaller than the number of training data sets of the first trained model. (Configuration 11) The information processing device according to any one of configurations 1 to 10, wherein the second trained model is a trained model obtained by additionally training the first trained model using skeletal information relating to at least one or more feature points of left and right elbow joints, wrists, fingers, the front and back of the hands, knee joints, ankles, toes, and the front and back of the feet. (Configuration 12) 12. The information processing device according to any one of configurations 1 to 11, further comprising a determination unit that determines whether or not there is consistency between the subject information and information included in the radiographic image capturing order. (Configuration 13) 13. The information processing device according to configuration 12, further comprising a display control unit that causes a display unit to display at least one of the optical image, skeletal information of the subject in the optical image, the subject region in the optical image, the subject information, and the determination result by the determination unit. (Configuration 14) 13. The information processing device according to configuration 12, further comprising a display control unit that displays a warning on a display unit when the determination unit outputs a determination result indicating that the consistency is not achieved. (Configuration 15) 13. The information processing device according to any one of configurations 1 to 12, further comprising an annotation unit that arranges the object information on the optical image or the radiation image as annotation information. (Configuration 16) The information processing device according to configuration 15, further comprising a display control unit that causes a display unit to display at least one of the optical image, skeletal information of the subject in the optical image, the subject region in the optical image, the subject information, the annotation information, and an optical image or a radiation image on which the annotation information is arranged. (Configuration 17) The information processing device according to configuration 2, wherein the determination unit adjusts parameters of the rule-based processing based on skeletal information of the subject in the optical image, skeletal information obtained using an optical image obtained by capturing an image of a phantom, or skeletal information obtained by projecting three-dimensional coordinates of virtual subject data generated by a three-dimensional modeling tool onto two-dimensional image coordinates, and subject information acquired via an operation unit. (Configuration 18) 18. The information processing device according to claim 17, wherein the parameters include at least one of a threshold value for threshold processing related to selecting feature points of the subject in the optical image, a positional relationship between the feature points and an irradiation area to which radiation is irradiated, and a correspondence relationship between the feature points and at least one of a part and a side of the subject. (Configuration 19) an optical device for capturing an optical image of the subject; a radiation generating device and a radiation detector for performing radiation imaging of the subject; an information processing device according to any one of configurations 1 to 18; A radiography system comprising: (Method 1) Obtaining an optical image by imaging a subject at a radiation imaging site; Estimating skeletal information of the subject in the optical image by using the optical image as input data of a second trained model obtained by additionally training a first trained model that estimates skeletal information about the subject's skeleton using skeletal information about a skeleton different from the skeleton indicated by the skeletal information learned by the first trained model; determining object information including information on at least one of laterality and location of the object in the optical image using skeletal information of the object in the optical image; An information processing method including: (Program 1) A program that, when executed by a computer, causes the computer to perform each step of the information processing method described in Method 1.

[0129] Although the present invention has been described above with reference to embodiments and examples, the present invention is not limited to the above embodiments and examples. The present invention also includes inventions that have been modified within the scope of the present invention and inventions equivalent to the present invention. Furthermore, the above-described embodiments and examples can be combined as appropriate within the scope of the present invention. [Explanation of symbols]

[0130] 200: Information processing device, 201: Optical image acquisition unit (acquisition unit), 202: Skeleton estimation unit (estimation unit), 203: Subject information determination unit (determination unit)

Claims

1. an acquisition unit that acquires an optical image obtained by imaging a subject at a radiation image imaging site; an estimation unit that estimates skeletal information of the subject in the optical image by using the optical image as input data of a second trained model obtained by additionally training a first trained model that estimates skeletal information about the subject's skeleton using skeletal information about a skeleton different from the skeleton indicated by the skeletal information learned by the first trained model; a determination unit that determines object information including information on at least one of laterality and location of the object in the optical image by rule-based processing using skeletal information of the object in the optical image; Equipped with the estimation unit estimates, as skeletal information of the subject in the optical image, coordinates of a plurality of feature points of the subject in the optical image and probabilities that the plurality of feature points correspond to predefined feature points of the subject; The determination unit determining an irradiation area in the optical image that is irradiated with radiation; selecting feature points of the object in the optical image based on a positional relationship between the coordinates estimated by the estimation unit and the irradiation area; an information processing device that determines the object information using the selected feature points;

2. The information processing device according to claim 1 , wherein the determination unit selects feature points of the object in the optical image using a threshold value for the probability estimated by the estimation unit, and determines the object information using the selected feature points.

3. The information processing apparatus according to claim 1 , wherein the determination unit determines the irradiation area based on an irradiation area of ​​a collimator lamp in the optical image or a center of the optical image.

4. The information processing apparatus according to claim 1 , wherein the determination unit determines the irradiation region based on an arrangement of a radiation generation device that irradiates the radiation and an optical device that generates the optical image.

5. 2. The information processing device according to claim 1, wherein the estimation unit extracts a subject region in the optical image, and estimates skeletal information of the subject in the optical image using the subject region in the optical image as input data for the second trained model.

6. 2. The information processing device according to claim 1, wherein the estimation unit classifies the optical image into a class indicating whether the optical image includes the entire body, a head, a hand, or a foot of the subject, and selects and uses a second trained model corresponding to the class.

7. The information processing device according to claim 1 , wherein the second trained model is a trained model obtained by additional training using a number of training data sets that is smaller than the number of training data sets of the first trained model.

8. 2. The information processing device according to claim 1, wherein the second trained model is a trained model obtained by additionally training the first trained model using skeletal information regarding at least one or more feature points of left and right elbow joints, wrists, fingers, the front and back of the hands, knee joints, ankles, toes, and the front and back of the feet.

9. The information processing apparatus according to claim 1 , further comprising a determination unit that determines whether or not there is consistency between the subject information and information included in the radiographic image capturing order.

10. 10. The information processing device according to claim 9, further comprising: a display control unit that causes a display unit to display at least one of the optical image, skeletal information of the subject in the optical image, the subject region in the optical image, the subject information, and the determination result by the determination unit.

11. The information processing apparatus according to claim 9 , further comprising: a display control unit that displays a warning on a display unit when the determination unit outputs a determination result indicating that the consistency is not achieved.

12. The information processing apparatus according to claim 1 , further comprising an annotation unit that arranges the object information on the optical image or the radiation image as annotation information.

13. 13. The information processing device according to claim 12, further comprising a display control unit that causes a display unit to display at least one of the optical image, skeletal information of the subject in the optical image, the subject region in the optical image, the subject information, the annotation information, and an optical image or a radiation image on which the annotation information is arranged.

14. 2. The information processing device according to claim 1, wherein the determination unit adjusts parameters of the rule-based processing based on skeletal information of the subject in the optical image, skeletal information obtained using an optical image obtained by capturing an image of a phantom, or skeletal information obtained by projecting three-dimensional coordinates of virtual subject data generated by a three-dimensional modeling tool onto two-dimensional image coordinates, and subject information acquired via an operation unit.

15. 15. The information processing device according to claim 14, wherein the parameters include at least one of a threshold value for threshold processing related to selection of feature points of the subject in the optical image, a positional relationship between the feature points and an irradiation region to which radiation is irradiated, and a correspondence relationship between the feature points and at least one of a part and a laterality of the subject.

16. An acquisition unit that acquires an optical image obtained by imaging a subject at a radiation image imaging site; an estimation unit that estimates skeletal information of the subject in the optical image by using the optical image as input data of a trained model that estimates skeletal information related to the subject's skeleton; a determination unit that determines object information including information on at least one of laterality and location of the object in the optical image by rule-based processing using skeletal information of the object in the optical image; Equipped with The determination unit determining an irradiation area in the optical image that is irradiated with radiation; selecting feature points of the object in the optical image based on a positional relationship between the coordinates of the feature points of the object in the optical image estimated by the estimation unit and the irradiation area; an information processing device that determines the object information using the selected feature points;

17. an optical device for capturing an optical image of the subject; a radiation generating device and a radiation detector for performing radiation imaging of the subject; An information processing device according to any one of claims 1 to 16; A radiography system comprising:

18. Obtaining an optical image by imaging a subject at a radiation imaging site; Estimating skeletal information of the subject in the optical image by using the optical image as input data of a second trained model obtained by additionally training a first trained model that estimates skeletal information about the subject's skeleton using skeletal information about a skeleton different from the skeleton indicated by the skeletal information learned by the first trained model; determining object information including information on at least one of laterality and location of the object in the optical image by rule-based processing using skeletal information of the object in the optical image; Including, the estimating includes estimating, as skeletal information of the subject in the optical image, coordinates of a plurality of feature points of the subject in the optical image and probabilities that the plurality of feature points correspond to predefined feature points of the subject; The determining step comprises: determining an irradiation area in the optical image that is irradiated with radiation; selecting feature points of the object in the optical image based on a positional relationship between the estimated coordinates and the irradiation area; An information processing method comprising determining the object information using the selected feature points.

19. Acquiring an optical image obtained by imaging a subject at a radiation imaging site; Estimating skeletal information of the subject in the optical image by using the optical image as input data of a trained model that estimates skeletal information related to the subject's skeleton; determining object information including information on at least one of laterality and location of the object in the optical image by rule-based processing using skeletal information of the object in the optical image; Including, The determining step comprises: determining an irradiation area in the optical image that is irradiated with radiation; selecting feature points of the object in the optical image based on the positional relationship between the coordinates of the feature points of the object in the optical image estimated by the estimation and the irradiation area; An information processing method comprising determining the object information using the selected feature points.

20. 20. A program that, when executed by a computer, causes the computer to execute each step of the information processing method according to claim 18 or 19.

Images