Medical work support system, work support method, and program

Abstract

To provide a medical work support system capable of detecting the abnormality of the position of a subject by a simple operation with a low load.SOLUTION: A medical work support system comprising: an image data acquisition unit that acquires image data captured in a state where a subject is placed on a bed; a placement region acquisition unit that acquires a placement region for placing the subject from the image data; a subject position data calculation unit that calculates position data of the subject from the image data; and an abnormality determination unit that determines an abnormality of a position of the subject based on the placement region acquired by the placement region acquisition unit and the position data of the subject.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a medical work support system and work support method for detecting a subject placed on a bed of a medical imaging device. [Background technology]

[0002] Large medical imaging devices that capture diagnostic images while controlling the position of a bed on which a patient rests, such as X-ray diagnostic devices, X-ray computed tomography (CT) devices, magnetic resonance imaging (MRI) devices, positron emission tomography (PET) devices, and single photon emission computed tomography (SPECT) devices, have many moving parts. Because operation of medical imaging devices requires careful attention to prevent contact between the moving parts of the medical imaging device and the patient, surrounding doctors, nurses, and testing equipment, a comprehensive safety confirmation method is desired.

[0003] One such safety confirmation method is a medical work support system that uses cameras installed in the examination room. Multiple cameras may be installed to reduce blind spots. Video data acquired from each camera is processed by an image processing device such as a computer to extract information necessary for safety confirmation, create a composite video, and assess the risk level. The results are then displayed on a monitor near the operator of the medical imaging device.

[0004] When checking the safety of subjects, the medical work support system must accurately detect the subject's body parts, checking the positions of the hands and arms, which are likely to come into contact with moving parts, and checking facial expressions to predict sudden behavior.

[0005] Patent Document 1 discloses an X-ray diagnostic device that has a function of detecting a subject by using a difference image between an image of a bed top taken when the subject is not lying down and an image of the bed top taken when the subject is lying down. That is, Patent Document 1 makes it possible to detect a person from a difference image between the image data acquired when a person is present and the reference image data by using a camera to capture reference image data when a person is present. Patent Document 2 discloses a method of arranging multiple cameras, selecting an appropriate one from among them, and displaying it for the operator, as well as an X-ray diagnostic device that has the function of switching to an appropriate camera depending on the inclination angle of a movable bed top in order to ensure the operator's field of vision. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 7118666 [Patent Document 2] Patent No. 5959972 Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 1, there was a problem that the reference image data had to be replaced when the lighting device or camera deteriorated over time or when the medical imaging device was moved, as the reference image data changed.

[0008] In Patent Document 2, the operator of the device has to pay attention not only to the camera image but also to the operation of the device, so even if an appropriate image can be displayed, there is a problem in that the burden of paying attention to the position of the subject is high. [Means for solving the problem]

[0009] Therefore, the present disclosure aims to provide a medical work support system that is capable of detecting abnormalities in the position of a subject with a low-load, simple operation.

[0010] The medical work support system according to the present disclosure includes: an image data acquisition unit that acquires image data captured with the subject placed on a bed; a placement area acquisition unit that acquires, from the image data, a placement area for placing the subject; a subject position data calculation unit that calculates position data of the subject from the image data; an abnormality determination unit that determines an abnormality in the position of the subject based on the placement area acquired by the placement area acquisition unit and the position data of the subject; The present invention is characterized by having the following. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to provide a medical work support system that is capable of detecting abnormalities in the position of a subject with a low-burden and simple operation. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a schematic diagram illustrating an example of a configuration of a medical work support system according to an embodiment of the present disclosure. [Figure 2] 4 is a flowchart showing an example of a processing procedure in a task assistance method according to the first embodiment. [Figure 3] 10 is a flowchart showing an example of processing for calculating a top board area in the task support method according to the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of arrangement of pre-set feature points. [Figure 5] FIG. 1 is a diagram for explaining learning data of a deep learning model. [Figure 6] FIG. 10 is a diagram for explaining the result of estimating the coordinates of feature points. [Figure 7] FIG. 10 is a diagram for explaining the result of estimating the coordinates of feature points. [Figure 8] FIG. 10 is a diagram illustrating acquisition of boundary parameters. [Figure 9] 10 is a flowchart showing an example of processing in a tabletop area extraction step in the task support method according to the first embodiment. [Figure 10] FIG. 10 is a diagram for explaining parameters for cutting out a top plate region. [Figure 11] 5A to 5C are diagrams for explaining a method of determining an abnormality in the work assistance method according to the first embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of a method for notifying of the possibility of interference. [Figure 13] FIG. 2 is a diagram for explaining an example of a method for notifying of the possibility of interference according to the first embodiment. [Figure 14] FIG. 2 is a diagram for explaining an example of a method for notifying of the possibility of interference according to the first embodiment. [Figure 15] FIG. 10 is a diagram for explaining a method for determining an abnormality according to a modified example of the first embodiment. [Figure 16] FIG. 10 is a diagram for explaining the influence of camera parallax. [Figure 17] 10 is a flowchart showing an example of a processing procedure in a task assistance method according to a second embodiment. [Figure 18] 10A and 10B are diagrams for explaining a method of determining an abnormality in a work assistance method according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, each configuration of the present disclosure will be described in detail using exemplary embodiments of the present disclosure with reference to the drawings. Note that in the drawings, similar or corresponding elements are denoted by the same reference numerals, and their description may be omitted or simplified.

[0014] In the following discussion, when reference is made to particular directions, such as left, right, front, back, up, and down, etc., it should be understood that such directions are described from the perspective of a user facing the system described below during exemplary operation.

[0015] First Embodiment A first embodiment according to the present disclosure will be described with reference to Figures 1 to 15. Here, an example in which the medical imaging apparatus is an X-ray diagnostic apparatus will be shown.

[0016] 1 is a schematic diagram showing an example of the configuration of a medical work support system according to the first embodiment. The medical work support system 100 according to the present disclosure comprises a camera 101, an image processing device 102 included in a personal computer (PC) 103, and a notification device 140 that notifies the user of abnormalities in the position of a subject.

[0017] The camera 101 is fixed to the ceiling of the examination room so as to capture an image of a placement area for placing the subject 120 and the environment including the subject 120. Here, the placement area is an area corresponding to the range in which the placed subject 120 should be accommodated, and is typically the area of ​​the tabletop 110 or bed 111. The camera 101 is, for example, an optical camera. Multiple cameras 101 may be installed to reduce blind spots. After the medical work support system 100 is started, the camera 101, image processing device 102, and PC 103 can communicate with each other via a network 105.

[0018] The image processing device 102 includes an image data acquisition unit 106 , a placement area acquisition unit 107 , a subject position data calculation unit 108 , and an abnormality determination unit 109 .

[0019] The image data acquisition unit 106 acquires image data captured with the subject 120 placed on the bed 111. The placement area acquisition unit 107 acquires a placement area for placing the subject 120 from the image data acquired by the image data acquisition unit 106. The subject position data calculation unit 108 calculates position data of the subject 120 from the image data acquired by the image data acquisition unit 106. Furthermore, the abnormality determination unit 109 determines an abnormality in the position of the subject 120 based on the placement area acquired by the placement area acquisition unit 107 and the subject position data.

[0020] The PC 103 including the image processing device 102 has a CPU (Central Processing Unit). The CPU performs predetermined operations according to programs stored in a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disk Drive), or the like provided in the PC 103 or another device functioning via the network 105. The processing performed by the CPU may include control of each function of the PC 103, as well as each processing performed by the image data acquisition unit 106, the placement area acquisition unit 107, the subject position data calculation unit 108, and the abnormality determination unit 109. Note that each of the above processing may also be executed by an MPU (Micro Controller Unit) instead of the CPU.

[0021] Here, an example is shown in which the image processing device 102 is included in the PC 103, but it may be configured with a workstation, a server, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a microcomputer, etc. It may also be configured with a tablet PC or smartphone integrated with the notification device 140.

[0022] The notification device 140 notifies the user of an abnormality determined by the abnormality determination unit 109. Specifically, the notification device 140 notifies the user of the determination result by the abnormality determination unit 109 regarding the possibility of interference between the subject 120 and the X-ray diagnostic device 104. The notification device 140 is, for example, a display that displays video or a speaker that notifies audio. The notification device 140 may be any device that can notify the user of the possibility of interference with the subject 120. In this embodiment, an example will be described in which the notification device 140 is a display that displays video.

[0023] The following describes an X-ray diagnostic device 104 having a placement area that is the subject of imaging by the camera 101. The X-ray diagnostic device 104 includes a tabletop 110, a bed 111, and an X-ray tube 112. In FIG. 1, the shorter side of the tabletop 110 is indicated by x. T The direction perpendicular to the surface of the tabletop 110 on which the subject 120 is placed is called the y T direction, and the longitudinal direction of the tabletop 110 is z T The direction.

[0024] In a major examination such as an upper gastrointestinal examination, the subject 120 is moved in the longitudinal direction (z T The patient lies on a tabletop 110 along the x direction. T The X-ray tube 112 and the support 113 move along the y direction. T direction and z T The tabletop 110, the bed 111, the X-ray tube 112, and the support 113 move in the x direction. T The X-ray diagnostic apparatus 104 rotates around an axis along the direction, and can change the posture of the subject 120 from a lying position (supine position) to a standing position (standing position). In other words, the placement area for placing the subject 120 is movable. As such, the X-ray diagnostic apparatus 104 has many moving parts, and therefore the user operating the X-ray diagnostic apparatus 104 needs to predict the behavior of the subject 120 while observing their condition, and ensure their safety.

[0025] The flow of processing performed in the medical work support system 100 will be described. The camera 101 captures an image with an angle of view that includes the placement area and the subject 120, and transmits the acquired camera image data frame by frame to an image processing device 102 built into the PC 103. The image processing device 102 detects, for example, the tabletop 110, which is the placement area, and the head of the subject 120 from the transmitted camera image data, and generates display image data by cutting out and combining each of them, as well as display image data indicating the possibility of interference between the subject 120 and the X-ray diagnostic device 104. The display image data is displayed on the notification device 140 frame by frame via the PC 103.

[0026] A user operating the X-ray diagnostic apparatus 104 looks at the display screen of the notification device 140 and operates the X-ray diagnostic apparatus 104 while checking the safety of the subject 120 .

[0027] A detailed processing procedure performed by the medical work support system 100 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of the processing procedure of the work support method in this embodiment.

[0028] The work support method according to this embodiment includes an image data acquisition step S202, a placement area acquisition step, a subject position data calculation step S205, and an abnormality determination step S206, and the placement area acquisition step includes a top area estimation step S203 and a top area extraction step S204. In this embodiment, an example will be described in which the work support method further includes a connection step S201, a notification step S207, and an end determination step S208.

[0029] The connection step S201 is a step of opening an input stream of camera image data sent from the camera 101 and an output stream for sending display image data to the PC 103.

[0030] The image data acquisition step S202 is a step in which the image data acquisition unit 106 acquires the latest camera image data sent from the input stream. That is, the image data acquired by the image data acquisition unit 106 is generated by the camera 101 capturing an image of the environment including the placement area and the subject 120.

[0031] As described above, the X-ray diagnostic apparatus 104 has many moving parts, and therefore, if the subject 120 puts his / her hands outside the tabletop 110 or grabs the edge of the tabletop 110 with his / her fingers while the apparatus is running, there is a risk of contact or getting caught. To detect such a situation, in the abnormality determination step S206, camera image data is input to determine whether or not there is a possibility that the subject 120 will interfere with the X-ray diagnostic apparatus 104.

[0032] The image processing device 102 performs the image data acquisition step S202, the top area estimation step S203, the top area extraction step S204, the subject position data calculation step S205, and the abnormality determination step S206.

[0033] The placement area acquisition step is a step of acquiring, from the image data, a placement area for placing the subject 120. That is, the placement area is an area corresponding to the range in which the placed subject 120 should be accommodated, and the medical work support system 100 detects abnormalities based on whether or not the subject 120 is accommodated in the placement area. Below, an example will be described in which, in the placement area acquisition step, the placement area acquisition unit 107 identifies, from the image data, a plurality of feature points located on the boundary of the placement area.

[0034] In this embodiment, an example will be described in which the placement area is a subject placement surface of a tabletop 110 provided on a bed 111. Note that the placement area can also be the subject support surface of the bed 111 (the surface on which the tabletop 110 is provided).

[0035] In the tabletop area estimation step S203, the placement area acquisition unit 107 acquires parameters that characterize the boundary of the tabletop 110 from the camera image data captured by the image processing device 102. The detailed processing procedure of the tabletop area estimation step S203 will be described using the flowchart in Fig. 3. In this embodiment, an example is shown that includes a heat map acquisition step S301 using a machine learning model to estimate the tabletop area.

[0036] FIG. 4 shows a tabletop 110 and a bed 111 of an X-ray diagnostic apparatus 104, a subject 120, and a plurality of virtual feature points K set in advance at arbitrary positions around the tabletop 110. i (K1, K2, ...K N ) (N is the total number of set feature points). A plurality of black dots drawn around the periphery of the tabletop 110 represent a plurality of feature points K i represents.

[0037] In the heat map acquisition step S301, an arbitrary feature point K set around the tabletop 110 is extracted from the input image. i Here, an example will be described in which the placement area acquisition unit 107 identifies a plurality of feature points using a machine learning model. i The coordinates of can be estimated using a deep learning model that outputs an image, such as an autoencoder.

[0038] In order for a deep learning model to learn, learning data is required. The learning data here refers to learning image data that shows the tabletop 110 and the feature points K i The annotation data indicates the coordinates of the feature points K on the image. The learning image data and annotation data can be obtained using, for example, computer graphics (CG). By using CG, it is possible to create a wide variety of learning image data and also to identify the feature points K on the image. i The coordinates can be obtained accurately.

[0039] Another way to obtain training data is to photograph training image data of the tabletop 110 with a camera and simultaneously attach an arbitrary marker such as an AR marker to the tabletop 110 to directly determine the position and orientation, thereby obtaining the training data.

[0040] The learning image data is array data of size H×W×C, where the height of the image of the tabletop 110 is H, the width is W, and the number of color channels is C (1 for a grayscale image, 3 for an RGB color image).

[0041] In the case of learning image data of the size mentioned above, the annotation data used for learning can be, for example, array data of H x W x N. Here, the number of channels, N, is the total number of virtual feature points mentioned above. In other words, the annotation data is a two-dimensional array with the same height and width as the learning image data, with feature points K i Each channel of the annotation data, 1 to N, has feature points K1 to K N represents the position of

[0042] Figure 5(a) shows the feature point K i 5(a) and 5(b) are diagrams showing the position of the i-th feature point superimposed on the training image data, and FIG. 5(b) is a diagram showing the annotation data (channel i of the annotation data) corresponding to the i-th feature point as an image.

[0043] Feature points K in the training image data i The coordinates of (x i , y i ), the channel i of the annotation data is at the coordinate (x i , y i ) is a peak and decays according to a Gaussian distribution (heat map). A heat map is a two-dimensional array with values ​​from 0 to 1, where the peak value is 1 and values ​​at positions far enough away from the peak are 0. Here, the values ​​included in the heat map are i The annotation data corresponds to the probability of the existence of such feature points K i It can be said to be a collection of heat maps centered on the position of

[0044] By converting annotation data into a heat map as described above, the accuracy of learning by a deep learning model can be improved.

[0045] The height and width of the annotation data array size can be changed to any size as long as the number of channels, N, is fixed. For example, by setting the array size of the annotation data to H / 2, W / 2, or N, the number of parameters in the deep learning model can be reduced compared to when the height and width are the same as those of the training image data. This is expected to reduce the time required to train the model and the calculation time during inference.

[0046] In the heat map acquisition step S301, the feature points K i For each (i=1~N), the feature point K i The existence probability of each feature point is obtained as a heat map. This results in an H×W×N array data. Here, the array data for channel i (i=1 to N) is i The larger the value at each coordinate in the sequence data, the higher the probability that a feature point exists at that coordinate.

[0047] In the heat map acquisition step S301, the feature point K i The image for estimating the existence probability of can be the camera image acquired in the image data acquisition step S202. Alternatively, the camera image can be subjected to any image processing, such as a linear processing filter such as edge enhancement or a non-linear processing filter such as a median noise reduction filter, and the like.

[0048] Next, in the peak detection step S302, peak detection is performed from each channel using an arbitrary method, and the feature point K i Estimate the coordinates on the image.

[0049] The feature points whose coordinates can be estimated by peak detection are N feature points K i (i=1 to N) may be a part of the coordinates of the feature points. Specifically, for example, it may not be possible to predict the coordinates of a feature point located on the back side of an object relative to the camera 101, or a feature point hidden by other objects such as the body of the subject 120 or a nearby doctor or nurse.

[0050] Figures 6 and 7 show heat maps of high-confidence feature points and low-confidence feature points, respectively.

[0051] In peak detection in the peak detection step S302, a threshold value can be set to determine whether the value indicated by the coordinate is a peak or not. For example, suppose the threshold value of the reliability for determining a peak is 0.4. Furthermore, suppose that as a result of peak detection, the coordinate value indicating the peak value is 0.9 in FIG. 6 and 0.3 in FIG. 7. In this case, the coordinate of the feature point of that index is estimated in FIG. 6, but the coordinate of the feature point of that index is not estimated in FIG. 7.

[0052] Below, feature point K from the image i The estimated position of the feature point P j (1≦j≦N), and let J be the set of indices j of the estimated feature points.

[0053] In the feature point grouping step S303, a set J of feature points that can be inferred from the peaks detected from the heat map in the peak detection step S302 is grouped. Since the tabletop 110 has a substantially rectangular shape, the feature points K set around the tabletop 110 are i It is possible to define which of the four sides each of the four points belongs to. i If there is a feature point K i can be treated as belonging to both of the two intersecting edges.

[0054] In the boundary parameter acquisition step S304, parameters indicating the four sides that characterize the boundary of the tabletop 110 are acquired. For example, as shown in FIG. 8, each of the four sides of the tabletop 110 can be approximated by a linear equation of the form y=ax+b in the coordinate system within the camera image. The parameters (a, b) are calculated based on the grouping performed in the feature point grouping step S303 by dividing a plurality of feature points K belonging to the same group. iIt can be found by the least squares method using the coordinates of

[0047] When the boundary line of the tabletop 110 is approximated by a linear expression in this way, the number of parameters is 2 x 4 sides = 8 parameters are estimated.

[0055] When the four sides of the tabletop 110 are viewed in a camera image, they are not actually straight due to the influence of distortion (distortion aberration) caused by the aberration of the camera lens. Therefore, by approximating with a polynomial of degree two or higher in the boundary parameter acquisition step S304, a more accurate boundary of the tabletop 110 can be obtained. To avoid overfitting, it is desirable to keep the degree of the polynomial at most about third degree.

[0056] Another method is to correct distortion in camera images by estimating internal parameters through camera calibration. For example, a known method involves capturing images of a checkerboard or similar object from multiple angles and estimating internal parameters from corresponding points in the three-dimensional space and the camera images. Another known method involves detecting straight lines in a camera image and estimating distortion coefficients that minimize the distortion of those lines. By obtaining a camera image in which distortion has been corrected using previously acquired internal parameters, the four sides of the tabletop 110 appear approximately straight in the camera image, making it possible to estimate physically reasonable parameters by approximating them with linear equations.

[0057] In the top board region cutting out step S204, the placement area obtaining unit 107 performs processing to cut out an area including the top board 110 as a rectangular region of interest (ROI). A detailed flow of the top board region cutting out step S204 is shown in FIG.

[0058] The center of gravity and center line direction estimation step S401 estimates the center of gravity position of the tabletop 110 in the camera image and a direction vector (hereinafter referred to as the center line direction) corresponding to the longitudinal direction of the tabletop 110. For example, the center of gravity position of the tabletop 110 is estimated from the plurality of feature points K obtained in the peak detection step S302. i The estimated feature points K iWhen there are M, the center of gravity w is given by the following equation (1) using coordinates xi (i=0, 1, ..., M-1).

number

[0059] The definition of the center line will be explained using Fig. 10. Fig. 10 is a diagram showing the tabletop 110 in a camera image and the straight lines L1 and L2 including the approximation line corresponding to the long side of the tabletop 110. As shown in Fig. 10, when the inclinations of the straight lines L1 and L2 including the long side are a1 and a2, respectively, the inclination a of the center line Lc is c is defined as the following equation (2).

number

[0060] Next, in the affine transformation step S402, the feature point K i The affine transformation of the coordinates of the center line Lc defined by equation (2) is performed. c Therefore, the angle θ between the vertical direction of the image and Lc is given by the following equation (3).

number

[0061] The x obtained above i , θ, it is possible to obtain an affine transformation matrix A of 3 rows and 3 columns shown in the following equation (4) such that the center line Lc is perpendicular to the center of gravity w of the tabletop 110.

number

[0062] In the clipping step S403, an area including the tabletop 110 is clipped as a placement area from the image after the affine transformation. The size of the clipping area is determined by, for example, the number of feature points K iIt can be determined by converting the coordinates of the points and specifying a rectangle that contains them all.

[0063] In the subject position data calculation step S205, the subject position data calculation unit 108 calculates position data of the subject 120 on the tabletop 110. The position data of the subject 120 is, for example, the coordinates of anatomical parts (landmarks) of the subject 120. The landmarks may cover the entire body of the subject 120, or may be extracted as the positions of parts of the body of the subject 120. Alternatively, detailed posture information of the subject 120 may be extracted, which is generated by identifying each landmark of the subject 120 and combining the landmarks.

[0064] One method for calculating subject position data is to use a machine learning model. Specifically, a trained deep neural network (hereinafter referred to as DNN) can be used as the machine learning model. The trained DNN can be a model that outputs the position of a target part of the subject 120 (such as a limb) using bounding box or binary mask information, or a model that outputs the coordinates of multiple landmarks and posture information of the subject 120.

[0065] Since the subject position data is obtained in the coordinate system of the image in which the top area is clipped, it is converted to the coordinate system before clipping and affine transformation using the following equation (5).

number

[0066] In the abnormality determination step S206, the abnormality determination unit 109 determines whether there is an abnormality in the position of the subject 120 based on the placement area acquired in the placement area acquisition step and the position data of the subject 120. Specifically, in the abnormality determination step S206, the position data of the subject 120 on the tabletop 110 obtained in the subject position data calculation step S205 and the boundary parameters of the tabletop 110 obtained in the boundary parameter acquisition step S304 can be used. Then, based on the position data and the boundary parameters, it can be determined whether or not a body part of the subject 120 is within the tabletop 110, as the presence or absence of an abnormality.

[0067] Fig. 11 is a diagram for explaining the method of determining abnormality in this embodiment. Fig. 11 shows a situation in which a subject 120 is lying face up on a tabletop 110, and points 121 and 122 are landmarks indicating the positions of the left and right hands of the subject 120 obtained in the subject position data calculation step S205.

[0068] In Figure 11, the right hand of subject 120 is within the area of ​​tabletop 110, but the left hand is outside the area of ​​tabletop 110. If the image is divided into three areas, A, B, and C, as shown in Figure 11 by straight lines L1 and L2 that indicate the boundaries of tabletop 110, the following equations hold for the coordinates of points within each area. Note that the positive directions of the x-axis and y-axis are the rightward and downward directions of the image, respectively. Area A:y <a1x+b1かつy<a2x+b2 Area B: y>a1x+b1 and y <a2x+b2 Area C: y>a1x+b1 and y>a2x+b2

[0069] In FIG. 11, point 121 representing the left hand of subject 120 is in area A, and point 122 representing the right hand is in area B. That is, the coordinates of point 121 are (x l ,y l ), and the coordinates of point 122 are (x r ,y r ) and Point 121:y l <a1x l +b1 and y l <a2x l +b2 Point 122:y r >a1x r +b1 and y r <a2x r +b2 In this way, the abnormality determination unit 109 performs area determination using the boundary parameters within the image for coordinates of interest, and determines whether or not a part of the body of the subject 120 is in an area that may interfere with the device.

[0070] Specifically, for example, the abnormality determination unit 109 may be configured to determine whether coordinates indicating a part of the body of the subject 120 (for example, points 121 and 122 in the above example) based on the position data of the subject 120 are inside or outside the placement area. Furthermore, for example, when coordinates indicating a part of the body of the subject 120 based on the position data of the subject 120 are outside the placement area, the abnormality determination unit 109 may be configured to calculate the distance between the coordinates and the placement area.

[0071] In a notification step S207, the notification device 140 notifies the user of the abnormality determined in the abnormality determination step S206. Specifically, if a part of the body of the subject 120 is in an area that may interfere with the X-ray diagnostic device 104, the notification device 140 notifies the user who operates the X-ray diagnostic device 104.

[0072] 12 to 14 are diagrams showing an example of a method for notifying of the possibility of interference using camera images. Fig. 12(a), Fig. 13(a), and Fig. 14(a) all show images displayed on the notification device 140 when there is no possibility of interference. Fig. 12(b), Fig. 13(b), and Fig. 14(b) all show images displayed on the notification device 140 when the subject 120 has his / her left hand outside the tabletop 110, creating a possibility of interference (a state in which notification is required).

[0073] Fig. 12 is a diagram showing an example of notifying an abnormality by displaying a warning in the upper left corner of the screen of the notification device 140. In the example shown in Fig. 12, the notification device 140 displays "Safe" as shown in Fig. 12(a) when there is no possibility of interference with the subject 120, and displays "Warning" as shown in Fig. 12(b) when there is a possibility of interference with the subject 120.

[0074] FIG. 13 is a diagram showing an example of notifying an abnormality by displaying a warning on a pictogram displayed in the upper left corner of the screen of the notification device 140. When there is no possibility of interference with the subject 120, the notification device 140 displays an image corresponding to the image data, in which pictograms representing each part of the subject 120 are uniformly superimposed, as shown in FIG. 13(a). The notification device 140 is configured to highlight and display on the pictogram the part of the subject 120 corresponding to the abnormality determined by the abnormality determination unit 109. Specifically, for example, when the subject 120 has his / her left hand outside the tabletop 110, the notification device 140 changes the color of the part corresponding to the left hand of the subject 120, as shown in FIG. 13(b).

[0075] The notification device 140 may be configured to display an image in which an effect that highlights the part of the subject 120 corresponding to the abnormality determined by the abnormality determination unit 109 is superimposed on an image corresponding to the image data so that the part can be identified. 14 is a diagram showing an example in which a possible interference area of ​​the subject 120 captured on the notification device 140 is highlighted with a bounding box. If there is no possibility of interference with the subject 120, the camera image is displayed as is on the notification device 140, as shown in FIG. 14(a). On the other hand, if there is a possibility of interference with the subject 120, a bounding box is overlaid on the left hand of the subject 120 on the notification device 140, as shown in FIG. 14(b).

[0076] The device or means for notifying the user in the above-described notification step S207 is merely an example, and it is possible to use other means for notifying the state of possible interference of the subject 120. For example, when there is a possibility of interference of the subject 120, a speaker (not shown) externally connected to the PC 103 may be used as a notification device to notify the user of the possibility of interference.

[0077] In the termination determination step S208, it is determined according to a predetermined criterion whether or not to terminate the operation of the medical work support system 100. If it is determined in the termination determination step S208 that the termination criterion is not met, the process returns to the image data acquisition step S202, and the steps up to the notification step S207 are repeated.

[0078] That is, until the termination criteria are met, the camera 101 can capture images of the environment including the placement area and the subject 120 multiple times over time. The image data acquisition unit 106 then sequentially acquires image data captured by the camera 101, and the abnormality determination unit 109 can sequentially determine abnormalities in the position of the subject 120 in parallel with the camera 101 capturing images over time.

[0079] If it is determined in the termination determination step S208 that the termination criteria are met, the work support method according to this embodiment is terminated.

[0080] The criteria for determining whether or not to terminate the process are not particularly limited and can be set appropriately, and may be, for example, whether or not an instruction to terminate the process is received from the user. In this case, the process of notifying the possibility of interference may be stopped in response to an instruction from the user. Furthermore, for example, the process may be linked to the operation of the X-ray diagnostic apparatus 104, and the termination may be determined based on whether or not the X-ray diagnostic apparatus 104 is turned on or off.

[0081] The exemplary configurations of the X-ray diagnostic apparatus 104 and the medical work support system 100 shown in this embodiment can also be modified to implement the work support method according to the present disclosure, and the medical work support system according to the present disclosure is not limited to those described above. For example, the camera 101 may be fixed to the X-ray diagnostic apparatus 104 or may be installed on a tripod or the like. Furthermore, although an X-ray diagnostic apparatus is used as an example of the medical imaging apparatus in this embodiment, any medical imaging apparatus, such as a CT or MRI, in which the apparatus main body and the subject 120 move relative to each other may be used.

[0082] As described above, the medical work support system 100 according to this embodiment can determine the possibility of interference of the subject 120 with the X-ray diagnostic apparatus 104 and notify the user through simple operations with low load. For example, the medical work support system 100 identifies the placement area and the subject position based on the same camera image data acquired during an examination, captured with the subject lying on a bed. This allows the medical work support system 100 to determine the possibility of interference, which is an abnormality in the position of the subject 120, and notify the user without adding work such as acquiring reference image data in advance.

[0083] <Modification 1 of the First Embodiment> In the first embodiment described above, the existence probability of feature points is estimated using a machine learning model in the tabletop region estimation step S203 in Fig. 2, but the present disclosure can also be implemented by other means. For example, by attaching AR markers, color markers, or the like to the four corners and predetermined positions around the tabletop 110 and detecting the marker positions from the acquired camera image, it is possible to detect the tabletop region using a method other than the method using machine learning.

[0084] <Modification 2 of the First Embodiment> 2, a pixel area corresponding to the tabletop 110 (hereinafter referred to as a tabletop pixel area) can be detected from the camera image data using a deep neural network (hereinafter referred to as a DNN). In this case, as a detection result, a binary image (hereinafter referred to as a tabletop mask) is output in which the value of pixels determined to correspond to the tabletop 110 is set to 1 and the other values ​​are set to 0.

[0085] Furthermore, in the subject position data calculation step S205, it is also possible to perform processing to detect a pixel area corresponding to the subject 120 from the camera image data using a deep neural network. In this case, as a detection result, a binary image (hereinafter referred to as a subject mask) is output in which the value of pixels determined to correspond to the subject 120 is set to 1 and the other values ​​are set to 0.

[0086] In the abnormality determination step S206, the camera image obtained in the image data acquisition step S202, the top mask, and the subject mask are input, and the possibility that the subject 120 will interfere with the X-ray diagnostic apparatus 104 is determined.

[0087] Fig. 15(a) is a diagram showing an example of a camera image, and Fig. 15(b) is a diagram showing a top plate mask 210 and a subject mask 220 generated from the camera image shown in Fig. 15(a). Region 221 in Fig. 15(b) represents a region inside the subject mask 220 but outside the top plate mask 210. In the abnormality determination step S206, the area (number of pixels) of region 221 is calculated, and if the calculated value is greater than a certain threshold, it can be determined that there is a possibility of interference.

[0088] Second Embodiment A second embodiment according to the present disclosure will be described.

[0089] An image captured by a single camera 101 is affected by parallax due to perspective projection. FIG. 16 shows an image of a subject 120 lying on a tabletop 110 with his left hand stretched out in front of his body, captured from a camera at a different angle. In the example image shown in FIG. 16(a), the left arm of the subject 120 is within the area of ​​the tabletop 110, but in the example image shown in FIG. 16(b), the left arm of the subject 120 appears to be outside the area of ​​the tabletop 110. If the camera 101 were installed in a position to capture the image shown in FIG. 16(b), it would be determined in the abnormality determination step S206 that the left arm of the subject 120 is outside the tabletop 110, and an alert would be sent to the user.

[0090] In this embodiment, the presence or absence of an abnormality is determined using three-dimensional position data of the tabletop 110 and the subject 120 based on the camera position.

[0091] 17 is a diagram showing the flow of the work support method according to this embodiment. The connecting step S201 to the top board area cutting step S204, the notifying step S207, and the end determining step S208 are the same as those in the first embodiment, and therefore will not be described here.

[0092] In the tabletop orientation estimation step S501, the placement area acquisition unit 107 estimates the position and orientation of the tabletop 110 in a coordinate system based on the camera 101. The PC 103 stores, in a memory (not shown), the feature points K i The 3D coordinate information of the feature point K is stored. i The 3D coordinate information of feature point K i unique index i (i=1~N) and feature points K i The individual points (K1, K2, …K N ) three-dimensional coordinate information (X i , Y i , Z i ) (i=1~N).

[0093] In estimating the position and orientation of the tabletop 110, the estimated feature point P obtained in the peak detection step S302 is j (1≦j≦N) coordinates in the camera image and feature point K iThe individual points (K1, K2, …K N ) three-dimensional coordinate information (X i , Y i , Z i ) and are used.

[0094] First, the feature point K i coordinate information (X i , Y i , Z i ), information on the feature point corresponding to the index J of the estimated feature point is read onto the work memory of the PC 103. The position and orientation of the tabletop 110 can be estimated by solving the Perspective-n-Point Problem (PnP problem) to find the external parameters (rotation vector, translation vector) of the camera 101. It is also possible to use a more advanced algorithm related to the PnP problem or an algorithm that removes outliers, such as Random Sample Consensus (RANSAC), in combination.

[0095] In the tabletop attitude estimation step S501, the position and attitude of the tabletop 110 may be obtained from drive information of the apparatus. The X-ray diagnostic apparatus 104 can be configured to be able to communicate with the medical work support system 100 via a network 105. The image processing apparatus 102 can then use information such as an attitude control command for the bed 111 of the X-ray diagnostic apparatus 104 and an examination order via the network 105.

[0096] Since the rotation axis of the bed 111 and the like are known from the design drawings of the X-ray diagnostic apparatus 104, the position and orientation of the bed 111 can be calculated directly from the orientation control command. By measuring the positional relationship between the camera 101 and the X-ray diagnostic apparatus 104 in advance, the position and orientation of the tabletop 110 in a coordinate system based on the camera 101 can be calculated.

[0097] As described above, in the tabletop orientation estimation step S501, the position and orientation of the tabletop 110 is obtained in a coordinate system based on the camera 101. As a result, the placement area acquisition unit 107 acquires the placement area as a three-dimensional range calculated as three-dimensional coordinates in a coordinate system based on the position where the image data was captured.

[0098] In the subject position data calculation step S205, two-dimensional coordinates within the camera image of a plurality of landmarks, which are anatomical features of the subject 120 within the camera image, and three-dimensional coordinates indicating the relative positional relationship of the landmarks within space are calculated.

[0099] In the subject posture estimation step S502, the position and posture of the subject 120 relative to the camera 101 are obtained by solving the PnP problem described above using the obtained two-dimensional and three-dimensional coordinates. That is, the three-dimensional coordinates of the landmarks of the subject 120 in a coordinate system based on the camera 101 are obtained.

[0100] As a result of the above, the subject position data calculation unit 108 calculates the position data of the subject 120 as three-dimensional coordinates in a coordinate system based on the position where the image data was captured.

[0101] In the abnormality determination step S206, the possibility of interference between the subject 120 and the X-ray diagnostic device 104 is determined based on the position and orientation of the tabletop 110 in a coordinate system based on the camera 101 and the three-dimensional coordinates of the subject's 120 landmarks.

[0102] In the abnormality determination step S206, a function representing a plane that includes the boundary of the tabletop 110 and is perpendicular to the surface on which the subject 120 lies, and the coordinates of the subject's 120's landmarks are used to determine whether or not the subject 120 is likely to interfere with the X-ray diagnostic device 104.

[0103] 18 is a diagram showing three-dimensional boundary surfaces D to G of the tabletop 110. By obtaining a set of equations f(x, y, z) = 0 that respectively represent the boundary surfaces D to G in a coordinate system based on the camera 101, it is possible to specify the four surfaces surrounding the tabletop 110 and the conditions within them.

[0104] If it is determined in the abnormality determination step S206 that the subject 120 may interfere with the X-ray diagnostic device 104, an alert is issued to the user by the notification device 140, a voice notification means (not shown), or the like in the notification step S207.

[0105] As described above, the medical work support system 100 according to this embodiment can determine the possibility of interference of the subject 120 with the X-ray diagnostic apparatus 104 without being affected by camera parallax, and notify the user.

[0106] It should be noted that the above-described embodiments merely illustrate specific examples of how the present disclosure can be implemented, and the technical scope of the present disclosure should not be interpreted as being limited by these embodiments. In other words, the present disclosure can be implemented in various forms without departing from its technical concept or main features. For example, it should be understood that embodiments in which part of the configuration of any embodiment is added to or substituted for part of the configuration of another embodiment are also embodiments to which the present disclosure can be applied.

[0107] Embodiments of the present disclosure include the following configurations and methods. (Configuration 1) an image data acquisition unit that acquires image data captured with the subject placed on a bed; a placement area acquisition unit that acquires, from the image data, a placement area for placing the subject; a subject position data calculation unit that calculates position data of the subject from the image data; an abnormality determination unit that determines an abnormality in the position of the subject based on the placement area acquired by the placement area acquisition unit and the position data of the subject; A medical work support system comprising: (Configuration 2) a camera that captures an image of an environment including the placement area and the subject; 2. The medical work support system according to configuration 1, wherein the image data is generated by capturing an image of the environment with the camera. (Configuration 3) the camera captures images of the environment multiple times over time; the image data acquisition unit sequentially acquires the image data captured by the camera, 3. The medical work support system according to configuration 2, wherein the abnormality determination unit sequentially determines abnormalities in the position of the subject in parallel with the time-lapse photography by the camera. (Configuration 4) The device further includes a notification device that notifies a user of the abnormality determined by the abnormality determination unit. 4. The medical work support system according to any one of configurations 1 to 3. (Configuration 5) 5. The medical work support system according to any one of configurations 1 to 4, wherein the placement area acquisition unit identifies a plurality of feature points located on the boundary of the placement area from the image data. (Configuration 6) 6. The medical work support system according to configuration 5, wherein the placement area acquisition unit identifies the plurality of feature points using a machine learning model. (Configuration 7) 7. The medical work support system according to any one of configurations 1 to 6, wherein the subject position data calculation unit calculates the position data using a machine learning model. (Configuration 8) 8. The medical work support system according to any one of configurations 1 to 7, wherein the placement area is a subject placement surface of a tabletop provided on the bed. (Configuration 9) 8. The medical work support system according to any one of configurations 1 to 7, wherein the placement area is a subject support surface of the bed. (Configuration 10) The medical work support system according to any one of configurations 1 to 9, wherein the abnormality determination unit is configured to determine whether coordinates indicating a part of the body of the subject based on the position data of the subject are inside or outside the placement area. (Configuration 11) The medical work support system according to any one of configurations 1 to 10, wherein the placement area acquisition unit acquires the placement area as a three-dimensional range calculated as three-dimensional coordinates in a coordinate system based on the position where the image data was taken. (Configuration 12) 12. The medical work support system according to any one of configurations 1 to 11, wherein the subject position data calculation unit calculates the position data of the subject as three-dimensional coordinates in a coordinate system based on the position where the image data was captured. (Configuration 13) 13. The medical work support system according to any one of configurations 1 to 12, wherein the placement area is movable. (Configuration 14) The medical work support system according to configuration 4, wherein the notification device is configured to display an image in which pictograms indicating each body part of the subject are superimposed on an image corresponding to the image data, and to highlight and display on the pictogram the body part of the subject corresponding to the abnormality determined by the abnormality determination unit. (Configuration 15) The medical work support system according to configuration 4, wherein the notification device is configured to display an image in which an effect that emphasizes the part of the subject corresponding to the abnormality determined by the abnormality determination unit so as to make it possible to identify the part is superimposed on an image corresponding to the image data. (Configuration 16) The medical work support system according to any one of configurations 1 to 15, wherein the abnormality determination unit is configured to calculate a distance between a coordinate indicating a part of the body of the subject based on the position data of the subject and the placement area when the coordinate is outside the placement area. (Configuration 17) 17. The medical work support system according to any one of configurations 1 to 16, wherein the bed is a bed for an X-ray diagnostic device. (Method 1) an image data acquisition step of acquiring image data captured with the subject placed on a bed; a placement area obtaining step of estimating a placement area for placing the subject from the image data; a subject position data calculation step of calculating position data of the subject from the image data; an abnormality determination step of determining an abnormality in the position of the subject based on the placement area acquired in the placement area acquisition step and the position data of the subject; A work support method comprising: (Configuration 18) A program for causing a computer to execute each step of the work support method described in Method 1. [Explanation of symbols]

[0108] 100 Medical Work Support System 101 Camera 102 Image processing device 103 PC 104 X-ray diagnostic equipment 105 Network 106 Image data acquisition unit 107 Placement area acquisition unit 108 Subject position data calculation unit 109 Abnormality determination section 110 Top Plate 111 Sleeper 112 X-ray tube 113 Post 120 subjects 140 Notification device

Claims

1. an image data acquisition unit that acquires image data captured with the subject placed on a bed; a placement area acquisition unit that acquires, from the image data, a placement area for placing the subject; a subject position data calculation unit that calculates position data of the subject from the image data; an abnormality determination unit that determines an abnormality in the position of the subject based on the placement area acquired by the placement area acquisition unit and the position data of the subject; A medical work support system comprising:

2. a camera that captures an image of an environment including the placement area and the subject; The medical work support system according to claim 1 , wherein the image data is generated by capturing an image of the environment with the camera.

3. the camera captures images of the environment multiple times over time; the image data acquisition unit sequentially acquires the image data captured by the camera, 3. The medical work support system according to claim 2, wherein the abnormality determination unit sequentially determines abnormalities in the position of the subject in parallel with the time-lapse photography by the camera.

4. The device further includes a notification device that notifies a user of the abnormality determined by the abnormality determination unit. The medical work support system according to claim 1 .

5. 5. The medical work support system according to claim 1, wherein the placement area acquisition unit identifies a plurality of feature points located on a boundary of the placement area from the image data.

6. The medical work support system according to claim 5 , wherein the placement area acquisition unit identifies the plurality of feature points using a machine learning model.

7. The medical work support system according to claim 1 , wherein the subject position data calculation unit calculates the position data using a machine learning model.

8. 5. The medical work support system according to claim 1, wherein the placement area is a surface of a tabletop provided on the bed where a subject is placed.

9. 5. The medical work support system according to claim 1, wherein the placement area is a patient support surface of the bed.

10. 5. The medical work support system according to claim 1, wherein the abnormality determination unit is configured to determine whether coordinates indicating a part of the subject's body based on the position data of the subject are inside or outside the placement area.

11. The medical work support system according to any one of claims 1 to 4, characterized in that the placement area acquisition unit acquires the placement area as a three-dimensional range calculated as three-dimensional coordinates in a coordinate system based on the position where the image data was taken.

12. 5. The medical work support system according to claim 1, wherein the subject position data calculation unit calculates the position data of the subject as three-dimensional coordinates in a coordinate system based on the position where the image data was captured.

13. 5. The medical work support system according to claim 1, wherein the placement area is movable.

14. 5. The medical work support system according to claim 4, wherein the notification device is configured to display an image in which pictograms indicating each part of the subject are superimposed on an image corresponding to the image data, and to highlight and display on the pictogram the part of the subject that corresponds to the abnormality determined by the abnormality determination unit.

15. The medical work support system of claim 4, wherein the notification device is configured to display an image in which an effect that emphasizes the part of the subject corresponding to the abnormality determined by the abnormality determination unit so as to make it possible to identify the part is superimposed on an image corresponding to the image data.

16. 5. The medical work support system according to claim 1, wherein the abnormality determination unit is configured to calculate a distance between a coordinate indicating a part of the subject's body based on the position data of the subject and the placement area when the coordinate is outside the placement area.

17. 5. The medical work support system according to claim 1, wherein the bed is a bed for an X-ray diagnostic apparatus.

18. an image data acquisition step of acquiring image data captured with the subject placed on a bed; a placement area obtaining step of estimating a placement area for placing the subject from the image data; a subject position data calculation step of calculating position data of the subject from the image data; an abnormality determination step of determining an abnormality in the position of the subject based on the placement area acquired in the placement area acquisition step and the position data of the subject; A work support method comprising:

19. A program for causing a computer to execute each step of the work support method according to claim 18.

Images