Monitoring camera system, radiography system and method

Abstract

To support efficient monitoring of a subject.SOLUTION: A monitoring camera system according to the embodiment includes a camera installed in an examination room that captures an environment including a medical device on which a subject can be placed, an image processing unit that generates display image data on the basis of camera image data obtained by the camera, and a display control unit that displays the display image data on a display unit. The image processing unit estimates a placement area corresponding to a region in which a subject is placed in the camera image data, generates corrected image data by correcting the camera image data according to the placement area, estimates a region of the subject included in the corrected image data, and generates display image data on the basis of the corrected image data and the region of the subject.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The embodiments disclosed in this specification and the drawings relate to a surveillance camera system, a radiation imaging system, and a method.

Background Art

[0002] Among medical devices that collect medical images or perform treatments on a subject, there are types that can place the subject. For example, the medical device includes a top plate placed on a hospital bed, and performs image collection or treatment on the subject placed on the top plate. At this time, in order to confirm the positional relationship between the medical device and the subject and the state of the subject, the subject is monitored using a camera installed in the examination room.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to assist in the efficient monitoring of a subject. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. The problems corresponding to the respective effects of each configuration shown in the embodiments described later can also be regarded as other problems.

Means for Solving the Problems

[0006] The monitoring camera system according to the embodiment includes a camera installed in an examination room for photographing an environment including a medical device on which a subject can be placed, an image processing unit that generates display image data based on the camera image data obtained by the camera, and a display control unit that causes the display unit to display the display image data. The image processing unit estimates a placement area corresponding to the area where the subject is placed in the camera image data, generates corrected image data obtained by correcting the camera image data according to the placement area, estimates the area of the subject included in the corrected image data, and generates display image data based on the corrected image data and the area of the subject.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

[0008] Hereinafter, embodiments of a surveillance camera system, a radiation imaging system, and a method will be described with reference to the drawings. In the following embodiments, parts denoted by the same reference numerals perform the same operations, and duplicate descriptions will be omitted as appropriate.

[0009] (First Embodiment) In the first embodiment, the system 1 in FIG. 1 will be described as an example. The system 1 includes a medical device 10, an information processing device 20, and a camera 30. The medical device 10, the information processing device 20, and the camera 30 are communicably connected via a network NW.

[0010] The medical device 10 is a device that is arranged in an examination room and performs collection of medical images and treatment on a subject. For example, the medical device 10 is a radiation diagnostic device such as an X-ray diagnostic device, a CT (Computed Tomography) device, a PET (Positron Emission computed Tomography) device, a SPECT (Single Photon Emission Computed Tomography) device, or a medical image diagnostic device such as a magnetic resonance imaging (MRI) device. Another example is that the medical device 10 is a treatment device such as a radiation treatment device. The medical device 10 performs collection of medical images and treatment on a subject placed on a top plate or a bed. That is, the medical device 10 is configured to be able to place a subject on it.

[0011] The information processing device 20 is a device that generates display image data based on camera image data obtained by the camera 30 and causes the display unit to display the display image data. For example, as shown in FIG. 1, the information processing device 20 includes a communication interface 21, an input interface 22, a display 23, a memory 24, and a processing circuit 25.

[0012] The communication interface 21 controls the transmission and communication of various data transmitted and received between the information processing device 20 and other devices and systems connected via the network NW. Specifically, the communication interface 21 is connected to the processing circuit 25, outputs data received from other devices or systems to the processing circuit 25, or transmits data output from the processing circuit 25 to other devices or systems. For example, the communication interface 21 is realized by a network card, a network adapter, a NIC (Network Interface Controller), or the like.

[0013] The input interface 22 receives various input operations from the user, converts the received input operations into electrical signals, and outputs them to the processing circuit 25. For example, the input interface 22 can be realized by a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touch pad that performs an input operation by touching the operation surface, a touch screen in which a display screen and a touch pad are integrated, a non-contact input circuit using an optical sensor, a voice input circuit, and the like. Note that the input interface 22 may be configured by a tablet terminal or the like that can communicate wirelessly with the information processing apparatus 20 main body. Further, the input interface 22 may be a circuit that receives an input operation from the user by motion capture. For example, the input interface 22 can receive the body movement and line of sight of the user as input operations by processing a signal acquired via a tracker or an image collected about the user. Further, the input interface 22 is not limited to those having physical operation components such as a mouse and a keyboard. For example, an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the information processing apparatus 20 and outputs this electrical signal to the processing circuit 25 is also included in the example of the input interface 22.

[0014] The display 23 displays various information. For example, the display 23 displays a GUI (Graphical User Interface) for receiving various instructions and settings from the user via the input interface 22. Further, the display 23 displays a composite image described later. For example, the display 23 is a liquid crystal display or a CRT (Cathode Ray Tube) display. The display 23 may be a desktop type or may be configured by a tablet terminal or the like that can communicate wirelessly with the information processing apparatus 20 main body.

[0015] The memory 24 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, a hard disk, an optical disk, etc. For example, the memory 24 stores various image data and programs for the circuits included in the information processing apparatus 20 to realize their functions. The memory 24 may also be realized by a server group (cloud) connected to the information processing apparatus 20 via the network NW.

[0016] The processing circuit 25 controls the operation of the entire information processing apparatus 20 by functioning as a control function 25a, an image processing function 25b, and a display control function 25c. For example, the processing circuit 25 functions as the control function 25a by reading out and executing a program corresponding to the control function 25a from the memory 24. Similarly, the processing circuit 25 functions as the image processing function 25b and the display control function 25c. The image processing function 25b is an example of an image processing unit. The display control function 25c is an example of a display control unit. Details of the processing by the processing circuit 25 will be described later.

[0017] In the information processing apparatus 20 shown in FIG. 1, each processing function is stored in the memory 24 in the form of a program executable by a computer. The processing circuit 25 is a processor that realizes the functions corresponding to the respective programs by reading out and executing the programs from the memory 24. In other words, the processing circuit 25 in the state of having read a program has the functions corresponding to the read program.

[0018] Note that in FIG. 1, the control function 25a, the image processing function 25b, and the display control function 25c are described as being realized by a single processing circuit 25. However, the processing circuit 25 may be configured by combining a plurality of independent processors, and each processor may realize its function by executing a program. Also, each processing function of the processing circuit 25 may be realized by being appropriately distributed or integrated into a single or a plurality of processing circuits.

[0019] Alternatively, the processing circuit 25 may implement functions by using the processor of an external device connected via the network NW. For example, the processing circuit 25 reads and executes a program corresponding to each function from the memory 24, and implements each function shown in FIG. 1 by using a server group (cloud) connected to the information processing apparatus 20 via the network NW as computing resources.

[0020] The camera 30 is installed in the examination room and photographs an environment including the medical device 10 on which the subject can be placed. Here, the environment including the medical device 10 is a range including at least a part of the medical device 10. For example, the camera 30 photographs, as the environment including the medical device 10, a range including the top plate provided in the medical device 10 and the subject placed on the top plate. For example, the camera 30 is installed on the wall surface or ceiling of the examination room.

[0021] In the system 1 of FIG. 1, the information processing apparatus 20 and the camera 30 constitute a surveillance camera system. The surveillance camera system composed of the information processing apparatus 20 and the camera 30 generates display image data for the subject to be imaged or treated by the medical device 10 and presents it to the operator of the medical device 10, thereby assisting the operator in monitoring the subject.

[0022] The overall configuration example of the system 1 has been described above. Next, a more specific example of the system 1 will be described by taking the case where the medical device 10 includes a radiation diagnostic apparatus as an example. The system 1 in the case where the medical device 10 includes a radiation diagnostic apparatus is also described as a radiation imaging system.

[0023] FIG. 2 is a diagram showing an example of the system 1 according to the first embodiment. The system 1 in FIG. 2 includes an IP camera 101, an image processing apparatus 102, a tablet PC 103, and a radiation diagnostic apparatus 104.

[0024] Among the various configurations shown in FIG. 2, the IP camera 101, the image processing device 102, and the tablet PC 103 constitute a surveillance camera system 100. The surveillance camera system 100 generates display image data for the subject 120 that is the object of image collection by the radiation diagnostic apparatus 104, and presents it to the operator of the radiation diagnostic apparatus 104, thereby assisting in the surveillance of the subject 120.

[0025] The IP camera 101 is an example of the camera 30. The IP camera 101 is a network camera to which an IP (Internet Protocol) is assigned and capable of data communication via a network. For example, after the surveillance camera system 100 is activated, the IP camera 101, the image processing device 102, and the tablet PC 103 can be connected to the network 106 via the wireless network hub 105 and communicate with each other. For example, the IP camera 101 is fixed to the ceiling 108 of the examination room so as to photograph the environment including the radiation diagnostic apparatus 104.

[0026] The radiation diagnostic apparatus 104, which is the imaging target of the IP camera 101, will be described. The radiation diagnostic apparatus 104 includes a top plate 110, a couch 111, and a radiation (X-ray) tube 112. In FIG. 2, an X-ray TV apparatus is illustrated as an example of the radiation diagnostic apparatus 104, but the embodiment is not limited thereto, and for example, other types of radiation diagnostic apparatuses such as an X-ray angiography apparatus can also be applied. The X-ray TV apparatus is used, for example, for upper gastrointestinal examinations.

[0027] In FIG. 2, the subject 120 is placed on the bed 111 and the top plate 110 in a lying state (lying position) along the longitudinal direction of the top plate 110 (the z-axis 132 of the top plate coordinate system). The top plate 110 can move along the x-axis 130 of the top plate coordinate system on the bed 111. Further, the X-ray tube 112 and the support column 113 can move along the y-axis 131 and the z-axis 132 of the top plate coordinate system. Further, the top plate 110, the bed 111, the X-ray tube 112, and the support column 113 are integrated and can rotate around the x-axis 130 of the top plate coordinate system to change the posture of the subject 120 from a lying state (lying position) to a standing state (standing position).

[0028] As described above, the radiation diagnostic apparatus 104 includes a plurality of movable parts and can perform various operations. When operating these movable parts, it is preferable for the operator of the radiation diagnostic apparatus 104 to predict the behavior while observing the state of the subject 120 to ensure safety. Also, regardless of whether the movable parts are operating or not, it is preferable for the operator to confirm the state of the subject 120. For example, there is a case where an endoscope is inserted into the subject 120 on the top plate 110, but there is also a case where the subject vomits due to the insertion of the endoscope, so it is preferable to confirm the expression of the subject 120.

[0029] However, it is not easy for the operator of the radiation diagnostic apparatus 104 to directly monitor the subject 120. For example, around the subject 120, the radiation diagnostic apparatus 104 and other inspection devices are arranged, and it may be difficult for the operator to confirm the subject 120 from the position of the operator. Also, there may be a plurality of medical personnel such as doctors and nurses in the examination room, and the view may be blocked by these medical personnel, making it difficult to confirm the subject 120. Also, when the operator is in the operation room outside the examination room, it becomes even more difficult to confirm the subject 120.

[0030] Therefore, the monitoring camera system 100 supports efficient monitoring of the subject 120 by generating and presenting display image data to the operator.

[0031] As an overview of the process, the surveillance camera system 100 generates display image data based on camera image data obtained by photographing an environment including the radiation diagnostic apparatus 104, and causes the display unit to display the display image data. For example, the IP camera 101 photographs an angle of view including the radiation diagnostic apparatus 104 to acquire camera image data, and transmits the acquired camera image data to the image processing apparatus 102 for each frame. The image processing apparatus 102 generates display image data from the transmitted camera image data. For example, when it is necessary to monitor the head of the subject 120, such as when the subject 120 may vomit, the image processing apparatus 102 detects the top plate 110 and the head of the subject 120, and generates display image data obtained by cutting out and synthesizing each of them. The display image data is transmitted to the tablet PC 103 for each frame and is displayed on the display 140. Thereby, the operator of the radiation diagnostic apparatus 104 can operate the radiation diagnostic apparatus 104 while checking the state of the subject 120 by looking at the display 140.

[0032] In FIG. 2, the image processing apparatus 102 includes a processing circuit and can execute an image processing function in the same manner as the processing circuit 25 shown in FIG. 1. That is, the image processing function of the image processing apparatus 102 generates display image data based on the camera image data obtained by the IP camera 101. The image processing function of the image processing apparatus 102 is an example of an image processing unit.

[0033] Further, the display 140 included in the tablet PC 103 is an example of a display unit. The tablet PC 103 includes a processing circuit and can execute a display control function in the same manner as the processing circuit 25 shown in FIG. 1. That is, the display control function of the tablet PC 103 causes the display 140 to display the display image data generated by the image processing apparatus 102. The display control function of the tablet PC 103 is an example of a display control unit. The combination of the image processing apparatus 102 and the tablet PC 103 shown in FIG. 2 is an example of the information processing apparatus 20.

[0034] The image processing device 102 generates display image data based on the camera image data obtained by the IP camera 101 (Camera 30). The processing procedure for generating the display image data performed in the image processing device 102 will be described with reference to FIG. 3. FIG. 3 is a flowchart showing an example of the processing procedure for generating the display image data.

[0035] The connection step S201 is a step of opening an input stream for the camera image data sent from the IP camera 101 and an output stream for transmitting the display image data to the tablet PC 103.

[0036] The camera image data reading step S202 is a step of reading the latest camera image data from the input stream.

[0037] The top plate pixel region detection step S203 is a step of detecting a pixel region corresponding to the top plate 110 (hereinafter referred to as the top plate pixel region) from the camera image data. In the flowchart of FIG. 3, the top plate 110 is an example of a placement region corresponding to the region where the subject 120 is placed. That is, the image processing device 102 estimates the placement region in the top plate pixel region detection step S203. For example, the image processing device 102 detects the top plate pixel region by a deep neural network (DNN) in the top plate pixel region detection step S203. That is, the placement region may be estimated by a deep neural network. For example, as a detection result, the DNN outputs a binary image (hereinafter referred to as a mask) in which the value of the pixel determined to correspond to the top plate 110 is "1" and the values of the other pixels are "0".

[0038] The top plate end polygon detection step S204 estimates a polygon (hereinafter referred to as the top plate end polygon) corresponding to the contour of the region where the pixel value of the mask is "1", and outputs the vertex coordinates of the polygon. The top plate end polygon is information indicating a region corresponding to the contour of the region where the pixel value of the mask is "1". The top plate end polygon can be paraphrased as the top plate end region. For example, the polygon can be obtained by detecting the edge of the mask through image processing and approximating the edge with a polyline using an algorithm such as the "Ramer-Douglas-Peucker" algorithm.

[0039] The centroid and center line direction estimation step S205 estimates the average coordinates of the pixels within the top plate pixel region (hereinafter referred to as the centroid) and the direction vector within the top plate pixel region corresponding to the longitudinal direction of the top plate (hereinafter referred to as the center line direction). When there are N pixels with a pixel value of "1" in the mask, the centroid w is given by the following formula (1) using the coordinates "xi (i = 0, 1,..., N - 1)" of the pixels with a pixel value of "1".

[0040]

Equation

[0041] The definition of the center line will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of the arrangement of the top plate 110 and the subject 120 in the camera image data 300. Although the top plate 110 in the three-dimensional space is generally rectangular, in the camera image data 300, due to the influence of oblique shooting, trapezoidal distortion may occur.

[0042] Hereinafter, the top plate 110 that appears as a two-dimensional figure in the camera image data 300 is also referred to as the top plate 301. The top plate 301 has a shape corresponding to the position of the IP camera 101 with respect to the top plate 110. For example, when the IP camera 101 is located on a straight line passing through the center of the top plate 110 and perpendicular to the placement surface of the top plate 110, the top plate 301 is rectangular. Also, when the IP camera 101 is located on a plane passing through the center of the top plate 110 and perpendicular to any of the four sides on the placement surface of the top plate 110, the top plate 301 is trapezoidal. When it does not fall into these types, the top plate 301 is a quadrilateral that does not include a pair of parallel sides. In FIG. 4, the top plate 301 is illustrated as a quadrilateral that does not include a pair of parallel sides.

[0043] Note that the subject 120 may also be affected by the oblique shooting. For example, although the subject 120 is lying on the top plate 110, in the camera image data 300 of FIG. 4, the body 303 and the legs 304 of the subject 120 protrude from the edge of the top plate 301.

[0044] As described above, the top plate 301 in FIG. 4 is distorted by the influence of trapezoidal distortion, and the two sides (side 305 and side 306) corresponding to the longitudinal direction of the top plate 110 are facing different directions. The center line 307 of the top plate 301 shown in FIG. 4 is a line that passes through the center of gravity 308 of the top plate 301 and faces the average direction of the side 305 and the side 306. The direction along the center line 307 is also referred to as the center line direction.

[0045] The estimation of the center line direction will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of a mask and a top plate edge polygon calculated for the camera image data 300. In the camera image data 300, the edge of the top plate 301 is partially hidden by the body of the subject 120. It is difficult to accurately estimate the pixels of the hidden top plate, and the shape of the region 321 with the pixel value "1" of the mask becomes inaccurate. Also, due to the influence of the image quality of the camera image data 300, the distortion at the edge of the screen, etc., even for the part of the edge of the top plate 301 that is not hidden by the body of the subject 120, it may not be extracted as a simple straight line. From the above, the top plate edge polygon 322 does not become a quadrilateral and is divided by a plurality of vertices, so it is rare to obtain the two sides (side 305 and side 306) in the longitudinal direction as they are. For example, in the case of the example shown in FIG. 5, the top plate edge polygon 322 is a nonagon, and side 305 is approximated by a broken line having one vertex, and side 306 is approximated by a broken line having two vertices.

[0046] Therefore, the image processing apparatus 102 selects, from among the plurality of sides of the top plate edge polygon 322, a side having a direction close to the two sides (side 305 and side 306) in the longitudinal direction, based on the length and direction, and takes the average of the directions of the two as the center line direction. For example, in the nonagonal top plate edge polygon 322 shown in FIG. 5, the front-side line segment of the two line segments corresponding to side 305 and the front-side line segment of the three line segments corresponding to side 306 can be selected, and the average of the directions of the two selected line segments can be taken as the center line direction.

[0047] In the top plate rotation clipping step S206, the image processing apparatus 102 generates corrected image data obtained by correcting the camera image data 300 according to the placement area. For example, the top plate rotation clipping step S206 is a step of rotating and cutting out a part of the camera image data 300 so that the center line direction is the vertical direction with the position of the center of gravity 308 of the mask as the center. In order to distinguish the cut-out pixel area from the top plate pixel area obtained at the time of top plate detection, it will be hereinafter referred to as the top plate cut-out area. The top plate cut-out area is an example of the corrected image data. That is, the image processing apparatus 102 may generate corrected image data (top plate cut-out area) by rotating the area including the placement area so that the center line of the placement area is in the vertical direction. In other words, the rotation angle at the time of generating the corrected image data is determined based on the direction of the side of the polygon (top plate edge polygon 322) surrounding the placement area.

[0048] The size of the top plate cut-out area can be any size including the top plate pixel area. That is, the image processing apparatus 102 may cut out a range that coincides with the top plate pixel area as the top plate cut-out area, or may cut out a range obtained by adding its peripheral area (margin) to the top plate pixel area as the top plate cut-out area. Further, the image processing apparatus 102 may rotate the top plate cut-out area cut out from the camera image data 300 so that the center line direction is the vertical direction, or may cut out the top plate cut-out area from the rotated camera image data 300.

[0049] The features of the top plate cut-out area will be described with reference to FIG. 6. In FIG. 6, as an example of the top plate cut-out area, a top plate cut-out area 330 is shown. The vertical direction of the top plate cut-out area 330 is the direction in which the center line 307 is estimated. That is, the top plate cut-out area 330 is image data obtained by rotating the area including the top plate pixel area so that the center line 307 is in the vertical direction. Although distortion remains, the shape of the top plate 301 is close to a vertically long and symmetric shape. The top head direction is not defined and may be reversed as in the example of FIG. 6.

[0050] The head detection step S207 is a step of detecting the head from the top plate cutting-out area 330. That is, in the head detection step S207, the image processing device 102 estimates the area of the head of the subject 120 included in the top plate cutting-out area 330. For example, in the head detection step S207, the image processing device 102 detects the position and size of the head by applying a DNN for head detection to the top plate cutting-out area 330.

[0051] Generally, the more limited the conditions of the input data are, the higher the accuracy of the DNN becomes. For example, the DNN for detecting the head from the image data of a subject in an upright posture has a higher head detection accuracy than the DNN for detecting the head from the image data of a subject facing in an arbitrary direction.

[0052] Since the top direction is not defined in the top plate cutting-out area 330, the image processing device 102 uses a DNN that is configured to detect the head from the image data of a subject in an upright or inverted posture to detect the head from the top plate cutting-out area 330. Alternatively, after the image processing device 102 detects the head from the top plate cutting-out area 330 using a DNN that is configured to detect the head from the image data of a subject in an upright posture, the image processing device 102 uses the DNN again to detect the head from the top plate cutting-out area 330 rotated 180 degrees, and adopts the result with a high evaluation value (accuracy) at the time of detection. In this way, by using the top plate cutting-out area 330 as the input to the DNN, the posture of the subject 120 is limited to an upright posture (or an inverted posture) that is easy for the DNN to detect, so that high-precision detection becomes possible.

[0053] The top plate vertical direction correction step S208 appropriately corrects the top plate cutting-out area 330 so that the top of the head is on the upper side. For example, when it is determined in the head detection step S207 that the head is on the lower side, the image processing device 102 rotates the top plate cutting-out area 330 by 180 degrees. On the other hand, when it is determined in the head detection step S207 that the head is on the upper side, the image processing device 102 omits the correction of the top plate cutting-out area 330 in the top plate vertical direction correction step S208.

[0054] The head clipping step S209 cuts out a pixel region centered on the position of the head (hereinafter referred to as the head cutout region) from the top plate cutout region 330. The image processing apparatus 102 may cut out the region determined as the head in the head detection step S207 as the head cutout region, or may cut out a range obtained by adding its peripheral region (margin) to the region determined as the head as the head cutout region.

[0055] In the display image data generation step S210, the image processing apparatus 102 generates display image data based on the corrected image data and the region of the subject. For example, the image processing apparatus 102 combines the camera image data, the top plate cutout region, and the head cutout region to generate display image data arranged as one piece of image data. Each piece of image data can also be combined with different sizes and angles.

[0056] Note that the top plate cutout region used in the display image data generation step S210 may be the top plate cutout region cut out in the top plate rotation clipping step S206, or may be separately cut out to generate the display image data. For example, in the top plate rotation clipping step S206, in order to avoid misrecognition of doctors and nurses in the vicinity, it is conceivable to cut off the periphery of the top plate. In this case, the image processing apparatus 102 regenerates top plate cutout regions of different sizes and uses them for generating the display image data.

[0057] The display image data writing step S211 writes the display image data to the output stream. For example, the tablet PC 103 displays the display image data on the display 140.

[0058] Also, the end determination S212 determines whether to stop generating the display image data in response to an instruction from the user. If it is determined in the end determination S212 not to stop generating the display image data (negative in the end determination S212), the surveillance camera system 100 returns to the camera image data reading step S202 and executes the above-described processing again. On the other hand, if it is determined to stop generating the display image data (positive in the end determination S212), the surveillance camera system 100 ends the processing. That is, the surveillance camera system 100 continues to generate the display image data based on the camera image data and display the display image data until it receives an end instruction from the user.

[0059] The flowchart of FIG. 3 is merely an example, and various modifications are possible. For example, in FIG. 3, the center line 307 is obtained based on the direction of the sides of the polygon (top plate edge polygon 322) surrounding the placement area, and an example of calculating the rotation angle in the top plate rotation clipping step S206 according to the direction of the center line 307 has been described. However, such rotation angle calculation processing does not necessarily need to be executed for all frames of the camera video data. For example, if there are patterns or components serving as marks on the top plate 110, the center of gravity and rotation angle calculated in the past frames may be slightly corrected based on the displacement of the marks. As an example, the rotation angle can be obtained based on the difference in the positions of an object commonly included in a plurality of camera image data with different shooting times. Thereby, the steps from the top plate pixel region detection step S203 to the center of gravity / center line direction estimation S205 can be omitted.

[0060] As described above, the surveillance camera system 100 according to the first embodiment includes an IP camera 101, an image processing device 102, and a tablet PC 103. The IP camera 101 is installed in the inspection room and photographs an environment including the radiation diagnostic apparatus 104 on which the subject 120 can be placed. The image processing device 102 generates display image data based on the camera image data 300 obtained by the IP camera 101. Further, the tablet PC 103 displays the display image data on the display 140.

[0061] Specifically, the image processing apparatus 102 estimates the top plate 110 as a placement area corresponding to the area where the subject 120 is placed in the camera image data 300. Further, the image processing apparatus 102 corrects the camera image data 300 according to the estimated top plate 110 and generates corrected image data. For example, the image processing apparatus 102 estimates the center line direction based on the estimation result of the top plate 110, and rotates and cuts out a part of the camera image data 300 so that the center line direction is in the vertical direction, thereby generating a top plate cut-out area 330. Further, the image processing apparatus 102 estimates the area of the subject 120 included in the corrected image data. For example, the image processing apparatus 102 applies a DNN for head detection to the top plate cut-out area 330 to detect the position and size of the head. Then, the image processing apparatus 102 generates display image data based on the corrected image data and the area of the subject 120.

[0062] With such a configuration, the monitoring camera system 100 according to the first embodiment can assist in efficient monitoring of the subject. For example, the monitoring camera system 100 can detect the head of the subject 120 without being affected by the posture change of the subject 120 by the radiation diagnostic apparatus 104, and can present the expression change of the subject 120 to the operator of the radiation diagnostic apparatus 104.

[0063] Generally, in the image data used for training the DNN, the humans included are often facing the same direction with the upper side of the image data as the top of the head. In contrast, in the case of the camera image data 300, although doctors and nurses are often standing upright, the posture of the subject 120 is various, and the subject 120 is often in a lying state (recumbent position). For this reason, even if the DNN is simply applied to the camera image data 300, the subject 120 may not be detected appropriately.

[0064] Also, for example, it is possible to make the DNN learn the postures that a plurality of humans in the examination room can take using the image data acquired by the monitoring camera in the examination room, such as the IP camera 101. However, training data needs to be created for each examination room, and the man-hour at the time of installing the monitoring camera system increases significantly.

[0065] In addition, for example, as in Patent Document 1, reference image data is captured by a surveillance camera in a state where there is no person, and a subject is detected as a difference image between the image data with a person and the reference image data, so that multiple people can be detected. However, it is necessary to re-acquire the reference image data when the lighting or the aging of the surveillance camera progresses, or when the inspection equipment moves.

[0066] On the other hand, in the surveillance camera system 100 according to the first embodiment, the DNN is applied to correction image data such as the top plate cutout region 330 rotated so that the center line direction of the top plate 110 is in the vertical direction, and the head is detected. As a result, without increasing operations such as the collection of training data and reference image data, the accuracy of detection by the DNN can be improved. Consequently, it becomes possible to present more appropriate display image data, and efficient surveillance of the subject 120 can be supported.

[0067] (Second Embodiment) In the second embodiment, an example of estimating the placement area using the information on the position and orientation of the camera will be described.

[0068] In the first embodiment, an example of detecting the top plate 110 as the placement area has been described. Hereinafter, as an example, an example of detecting the hospital bed 111 as the placement area will be described. That is, the placement area in the first embodiment is the top plate 110 of the medical device 10, and the placement area in the second embodiment is the hospital bed 111 of the medical device 10. Also, regarding the area of the subject 120 detected from the correction image data, there is no particular limitation, but as an example, the case of detecting the hand of the subject 120 from the correction image data will be described.

[0069] FIG. 7 is a diagram showing an example of the system 1 according to the second embodiment. The system 1 in FIG. 7 includes an IP camera 601, an IP camera 602, an image processing device 603, a display 604, and a radiation diagnostic device 104.

[0070] Among the various configurations shown in FIG. 7, the IP camera 601, the IP camera 602, the image processing device 603, and the display 604 constitute a surveillance camera system 600. The surveillance camera system 600 generates display image data for the subject 120 that is the object of image collection by the radiation diagnostic apparatus 104 and presents it to the operator of the radiation diagnostic apparatus 104, thereby assisting in the surveillance of the subject 120.

[0071] The IP camera 601 and the IP camera 602 are network cameras capable of data communication, similar to the IP camera 101, and are an example of the camera 30. For example, the IP camera 601 and the IP camera 602 are fixed to the ceiling 108 of the examination room so as to photograph the environment including the radiation diagnostic apparatus 104. For example, after the surveillance camera system 600 is activated, the IP camera 601, the IP camera 602, and the image processing device 603 can communicate with each other via the wired network hub 605. Further, the display 604 is connected to the image processing device 603 by a display cable 606 and displays the image sent from the image processing device 603. The display 604 is an example of a display unit.

[0072] The radiation diagnostic apparatus 104, which is the imaging target of the IP camera 601 and the IP camera 602, is different from the radiation diagnostic apparatus 104 described in the first embodiment in that spherical markers 610 and 611 are attached to the side surface of the hospital bed 111. Hereinafter, the marker attached to the side surface corresponding to the long side of the hospital bed 111 will be described as the marker 610, and the marker attached to the side surface corresponding to the short side will be described as the marker 611. For example, the IP camera 601 photographs the range including the marker 610 as the environment including the radiation diagnostic apparatus 104. Further, the IP camera 602 photographs the range including the marker 611 as the environment including the radiation diagnostic apparatus 104.

[0073] As an overview of the process, IP cameras 601 and 602 capture an angle of view including the radiation diagnostic apparatus 104 to obtain camera image data, and transmit the obtained camera image data to the image processing apparatus 603 for each frame. The image processing apparatus 603 generates display image data from the transmitted camera image data.

[0074] For example, when moving the top plate 110 along the x-axis 130 of the top plate coordinate system, it is preferable to place the hand of the subject 120 at a predetermined position so that the hand of the subject 120 is not pinched between the top plate 110 and the bed 111. More specifically, with respect to the hand of the subject 120, it is preferable to keep it in a state of being placed on the top plate 110 or in a state of holding a handle provided on the top plate 110. However, due to the movement of the subject 120, the hand may deviate from a predetermined position on the top plate 110. For example, although a plastic sheet may be laid between the top plate 110 and the bed 111, the subject 120 may grab this plastic sheet.

[0075] Thus, when it is necessary to monitor the hand of the subject 120, the image processing apparatus 603 detects the top plate 110 and the hand of the subject 120 from the transmitted camera image data, and generates display image data obtained by cutting out and synthesizing each of them. The display image data is displayed on the display screen 640 on the display 604 for each frame. Thereby, the operator of the radiation diagnostic apparatus 104 can operate the radiation diagnostic apparatus 104 while checking the state of the subject 120 by looking at the display 604. For example, the operator can move the top plate 110 along the x-axis 130 of the top plate coordinate system after confirming that the hand of the subject 120 is not involved in the movement of the top plate 110.

[0076] In FIG. 7, the image processing apparatus 603 includes a processing circuit and can execute an image processing function and a display control function in the same manner as the processing circuit 25 shown in FIG. 1. That is, the image processing function of the image processing apparatus 603 generates display image data based on the camera image data obtained by the IP camera 601 and the IP camera 602. Further, the display control function of the image processing apparatus 603 causes the display unit to display the display image data. The image processing function of the image processing apparatus 603 is an example of an image processing unit. The display control function of the image processing apparatus 603 is an example of a display control unit. The image processing apparatus 603 is an example of the information processing apparatus 20.

[0077] The measurement procedure of the camera posture performed after the camera is installed in the image processing apparatus 603 will be described with reference to FIG. 8. FIG. 8 is a flowchart showing an example of the measurement procedure of the camera posture.

[0078] The reference coordinate value reading step S701 is a step of reading the reference coordinate value from a file. The reference coordinate value will be described with reference to FIG. 9. The reference coordinate value is the three-dimensional coordinate value of the corner point 801 of the hospital bed 111 and the three-dimensional coordinate value of the center of the spherical marker 802, and is obtained, for example, from the design drawing of the hospital bed 111. Although the coordinate axes can be arbitrarily given, here, the x-axis 810, y-axis 811, and z-axis 812 having the origin 813 at the center of the upper surface of the hospital bed are provided such that the x-axis 810 is parallel to the short side direction of the hospital bed 111 and the z-axis 812 is parallel to the long side direction of the hospital bed 111.

[0079] The connection step S702 is a step of opening an input stream of the camera image data sent from the IP camera 601 and the IP camera 602.

[0080] The reference state restoration step S703 is a step of leveling the hospital bed 111. For example, the reference state restoration step S703 is executed by an operator operating the radiation diagnostic apparatus 104. Other parameters such as the height of the radiation (X-ray) tube 112 and the position of the top plate 110 can be arbitrarily determined, but it is necessary to ensure that the corner point 801 of the hospital bed 111 is not hidden in the camera image data.

[0081] The camera image data reading step S704 is a step of reading camera image data from an input stream.

[0082] The corner pixel position measurement step S705 is a step of obtaining the positions of pixels corresponding to the corners of the hospital bed from the camera image data. For example, the operator opens the camera image data using an arbitrary image viewer that can display the position of the pixel pointed to by the pointer, visually searches for the pixel corresponding to the corner from the screen of the image viewer, and records the position of the pixel in a file.

[0083] The camera position and orientation measurement step S706 estimates camera parameters using the pixel positions corresponding to the corners recorded in the file and the three-dimensional coordinate values of the corner points 801 included in the reference coordinate values. The camera parameters are the three-dimensional coordinates of the pinhole when the camera is approximated by a pinhole camera model, the direction vector of the optical axis, and the direction vectors of two orthogonal axes on the image sensor plane, and correspond to the position and orientation of the camera. As described in Non-Patent Document 1, using the pinhole camera model, the three-dimensional coordinates of the corner points 801 can be converted into pixel positions on the camera image data. By optimizing the values of the camera parameters so that the distance between the pixel positions obtained by the conversion and the pixel positions obtained by the corner pixel position measurement step S705 is close, the positions and orientations of the IP camera 601 and the IP camera 602 can be measured. The obtained camera parameters are saved in a file.

[0084] Next, the processing procedure for generating display image data performed in the image processing apparatus 603 will be described with reference to FIG. 10. FIG. 10 is a flowchart showing an example of the processing procedure for generating display image data.

[0085] The parameter reading step S901 is a step of reading the reference coordinate values and the camera parameters from a file.

[0086] The connection process S902 is a process of opening an input stream for camera image data sent from the IP camera 601 and the IP camera 602, and an output stream for sending display image data to the display 604.

[0087] The camera image data reading process S903 is a process of reading the latest camera image data from the input stream.

[0088] The marker detection process S904 is a process of detecting the center pixel position of the marker 610 from the camera image data of the IP camera 601, and detecting the center pixel position of the marker 611 from the camera image data of the IP camera 602. Since the shapes of the marker 610 and the marker 611 are spherical, the marker appears circular even in an oblique shot. Therefore, the center coordinates can be accurately calculated from three points corresponding to the outer periphery of the marker. The marker 610 and the marker 611 may be partially hidden by doctors, nurses, and inspection equipment around the hospital bed. However, if each of the marker 610 and the marker 611 contains one or more, and a total of three or more center pixel positions can be obtained, the calculation in the subsequent process can be executed. If not, the creation of the display image data for this camera image data is skipped, and the process is executed from the next camera image data reading process S903.

[0089] The hospital bed position and posture calculation process S905 is a process of calculating the position and posture parameters of the hospital bed 111 based on the center pixel positions of the marker 610 and the marker 611, and the three-dimensional coordinate values of the center of the spherical marker 802 included in the reference coordinate values. The position and posture parameters of the hospital bed 111 are the coordinates of the origin 813 after the posture change, and the coordinate axes (x-axis 810, y-axis 811, z-axis 812) after the posture change. Since the camera parameters are known by the camera position and posture measurement process S706, for example, as described in Non-Patent Document 1, the three-dimensional coordinates of the center of the spherical marker 802 can be converted into pixel positions on the camera image data. By optimizing the position and posture parameters of the hospital bed 111 so that the distance between the pixel positions obtained by the conversion and the center pixel positions of the marker 610 and the marker 611 becomes small, the position and posture of the hospital bed can be calculated.

[0090] The projective transformation calculation step S906 is a step of calculating parameters of a projective transformation that converts a pixel position corresponding to the corner point 801 on the camera image data (hereinafter referred to as the input corner pixel position) into an ideal pixel position with the longitudinal direction being the vertical direction (hereinafter referred to as the output corner pixel position). The input corner pixel position can be calculated based on the three-dimensional coordinate values of the corner point 801 and the position and orientation parameters of the hospital bed 111. The output corner pixel position can be arbitrarily selected, but it is necessary to match the aspect ratio of the rectangle formed by the corner point 801. The projective transformation is given by the following formula (2).

[0091]

Equation

[0092] In Equation (2), "x" and "y" are the positions of pixels on the camera image data, and "x'" and "y'" are the positions of pixels in the output destination image data. "p1" to "p8" are projective transformation parameters. Substituting four input corner pixel positions and four output corner pixel positions into the variables "x", "y", "x'", and "y'" in Equation (2) gives eight simultaneous equations with the projective transformation parameters as unknown variables. By solving this system of simultaneous equations, the projective transformation parameters can be obtained.

[0093] The bed clipping step S907 converts the camera image data by the projective transformation of Equation (2), and cuts out a pixel region (hereinafter referred to as the bed cut-out region) that includes a rectangle connecting the output corner pixel positions. The bed cut-out region is an example of corrected image data. Within the bed cut-out region, the bed becomes a vertically long rectangle. That is, the image processing device 603 may generate corrected image data (bed cut-out region) by converting a region including the placement region so as to approach a vertically long rectangle. The size of the bed cut-out region can be arbitrarily selected, and the peripheral portion of the bed can also be included. That is, the image processing device 603 may cut out a range that coincides with the rectangle connecting the output corner pixel positions as the bed cut-out region, or may cut out a range obtained by adding its peripheral region (margin) to the rectangle as the bed cut-out region. The vertical direction of the bed cut-out region can be changed when giving the output corner pixel positions in the projective transformation calculation step S906.

[0094] The hand detection step S908 is a step of detecting the hand of the subject 120 from the bed cut-out region. That is, in the hand detection step S908, the image processing device 603 estimates the region of the hand of the subject 120 included in the bed cut-out region. For example, the image processing device 603 applies a DNN for skeleton detection to the bed cut-out region to detect the position and size of the hand.

[0095] The hand clipping step S909 is a step of cutting out a pixel region (hereinafter referred to as the hand cut-out region) centered on the position of the hand from the bed cut-out region. The image processing device 603 may cut out the region determined as the hand in the hand detection step S908 as the hand cut-out region, or may cut out a range obtained by adding its peripheral region (margin) to the region determined as the hand as the hand cut-out region.

[0096] The step S910 of generating display image data is a step of generating display image data arranged as one piece of image data by combining camera image data, a bed cutout area, and a hand cutout area. Since there are multiple cameras, instead of arranging all of the various types of image data, it may be arranged in a combination specified by the user or a combination determined by judging visibility. The writing step S911 and the end determination S912 of the display image data are the same as the writing step S210 and the end determination S212 of the display image data shown in FIG. 3, and thus the description thereof is omitted.

[0097] As described above, the monitoring camera system 600 according to the second embodiment includes an IP camera 601, an IP camera 602, and an image processing device 603. The IP camera 601 and the IP camera 602 are installed in the examination room and photograph an environment including the radiation diagnostic apparatus 104 on which the subject 120 can be placed. The image processing device 603 generates display image data based on the camera image data obtained by the image processing device 603. Further, the image processing device 603 causes the display 604 to display the display image data.

[0098] Specifically, the image processing device 603 estimates the bed 111 as a placement area corresponding to the area where the subject 120 is placed in the camera image data based on a marker attached to the bed 111. Further, the image processing device 603 corrects the camera image data according to the estimated bed 111 and generates corrected image data. For example, the image processing device 603 performs projective transformation on the camera image data based on the estimation result of the bed 111, and generates a bed cutout area including a rectangle connecting the output corner pixel positions as the corrected image data. Further, the image processing device 603 estimates the area of the subject 120 included in the corrected image data. For example, the image processing device 603 applies a DNN for skeleton detection to the bed cutout area to detect the position and size of the hand. Then, the image processing device 603 generates display image data based on the corrected image data and the area of the subject 120.

[0099] With such a configuration, the monitoring camera system 600 according to the second embodiment can assist in the efficient monitoring of the subject 120. For example, the monitoring camera system 600 can accurately detect the posture-changing hospital bed 111 from the camera image data based on the markers attached to the hospital bed 111. That is, the monitoring camera system 600 can accurately estimate the placement area corresponding to the area where the subject 120 is placed.

[0100] In addition, the monitoring camera system 600 can improve the detection accuracy of the subject 120 by making the shape of the hospital bed 111 in the image data a vertically long rectangle. That is, by unifying the input data to the DNN into vertically long rectangular image data, the detection accuracy of the subject 120 by the DNN can be improved.

[0101] Although the example of detecting the hospital bed 111 as the placement area has been described, the embodiment is not limited thereto. For example, when markers similar to the markers 610 and 611 are attached to the top plate 110, the top plate 110 can be detected as the placement area based on the detection result of the markers and the positions and postures of the IP camera 601 and the IP camera 602. Also, as described above, the placement area may be the top plate 110. That is, the placement area may be estimated based on the detection result of the markers attached to the top plate 110 or the hospital bed 111 of the medical device 10 and the position and posture of the camera 30.

[0102] (Third Embodiment) In the second embodiment described above, the example of estimating the hospital bed 111 in the camera image data based on the markers attached to the hospital bed 111 has been described. In the third embodiment, an example of estimating the hospital bed 111 in the camera image data based on the control information of the hospital bed 111 in the radiation diagnostic apparatus 104 will be described.

[0103] FIG. 11 is a diagram showing an example of the system 1 according to the third embodiment. The system 1 shown in FIG. 11 can be basically configured in the same manner as the system 1 shown in FIG. 7, but the markers 610 and 611 can be omitted. Further, the radiation diagnostic apparatus 104 shown in FIG. 11 is connected to a wired network hub 605 by a network cable 107 and can communicate with the surveillance camera system 600. Thereby, the image processing apparatus 603 can utilize information such as an attitude control command for the stretcher 111 of the radiation diagnostic apparatus 104 and an examination order.

[0104] From the design drawing of the radiation diagnostic apparatus 104 and the like, the rotation axis of the stretcher 111 and the like are known. Therefore, the image processing apparatus 603 can directly calculate the position and attitude of the stretcher 111 from the attitude control command of the stretcher 111. For example, the image processing apparatus 603 can omit the marker detection step S904 and the stretcher position / attitude calculation step S905 shown in FIG. 10, and in the projective transformation step S906, can use the position and attitude of the stretcher calculated from the attitude control command.

[0105] Also, in the display image data generation step S910 shown in FIG. 10, the image processing apparatus 603 can change the combination method of various image data such as camera image data, the stretcher cut-out area, and the hand cut-out area according to the examination order obtained from the radiation diagnostic apparatus 104.

[0106] As described above, even when no marker is attached to the stretcher 111 or when the marker attached to the stretcher 111 does not appear in the camera image data, the surveillance camera system 600 according to the third embodiment can detect the stretcher 111 from the camera image data as a placement area, and thus can support efficient surveillance of the subject 120 by enabling generation of the display image data.

[0107] Although an example of detecting the bed 111 as the placement area has been described, the embodiment is not limited thereto. For example, when the control information of the top plate 110 is available, the top plate 110 can be detected as the placement area based on the control information and the positions and postures of the IP camera 601 and the IP camera 602. That is, the placement area may be estimated based on the control information of the top plate 110 or the bed 111 of the medical device 10 and the position and posture of the camera 30.

[0108] (Fourth Embodiment) In the fourth embodiment, a specific example of display control will be described. Although the specific configuration of the surveillance camera system according to the fourth embodiment is not particularly limited, as an example, it can be configured in the same manner as the surveillance camera system 100 shown in FIG. 2.

[0109] FIG. 12 shows a display example. For example, the tablet PC 103 causes the subject confirmation screen 1000 shown in FIG. 12 to be displayed on the display 140. The subject confirmation screen 1000 includes an image data drawing area 1001, a list box 1002 for layout selection, a full-screen display instruction button 1003, and a transition button 1004 to the layout setting screen.

[0110] The image data drawing area 1001 is an area for drawing the display image data sent from the image processing device 102. The list box 1002 for layout selection is used to select information regarding the configuration of the image data in the display image data (hereinafter referred to as layout information). For example, the layout information is stored in the image processing device 102 at the time of product design or registration on the layout setting screen described later, and a label based on the inspection content and the creator is attached. For example, in FIG. 12, "ERCP examination - Technician △△" is displayed in the list box 1002 as a label consisting of a combination of the inspection content and the creator.

[0111] FIG. 12 is an example of an object and a user interface for specifying a predetermined direction when generating the above-described rotated image data. That is, the tablet PC 103 can display an object and a user interface for specifying a predetermined direction on the display 140.

[0112] For example, the tablet PC 103 can receive input of the inspection content of the medical device 10 via the list box 1002. Here, by presetting an object and a predetermined direction for each of the inspection contents selectable from the list box 1002, an object and a predetermined direction corresponding to the selected inspection content can be specified. For example, as shown in FIG. 12, when "ERCP examination" is selected, "head" can be specified as the object and "vertical direction" can be specified as the predetermined direction. That is, the tablet PC 103 can specify an object and a predetermined direction by receiving input of the inspection content of the medical device 10 via the user interface.

[0113] Also, for example, the tablet PC 103 can receive input of the user information of the medical device 10 via the list box 1002. Here, by presetting an object and a predetermined direction for each of the user information selectable from the list box 1002, an object and a predetermined direction corresponding to the selected user information can be specified. For example, as shown in FIG. 12, when "Engineer XX" is selected, "head" can be specified as the object and "vertical direction" can be specified as the predetermined direction. That is, the tablet PC 103 can specify an object and a predetermined direction by receiving input of the user information of the medical device 10 via the user interface.

[0114] In addition, in FIG. 12, in the list box 1002, the inspection content and the user information are combined and displayed. That is, the tablet PC 103 can specify an object and a predetermined direction by receiving the input of the inspection content and the user information of the medical device 10 via the user interface. The tablet PC 103 may be configured to accept the input of the inspection content and the user information separately, or may be configured to accept the input of only one of them.

[0115] The label of the layout information selected by the list box 1002 is sent from the tablet PC 103 to the image processing device 102, and the configuration of the image data in the display image data is changed. For example, according to the layout of "ERCP examination - Technician △△", the camera image data 1011, the camera image data 1012 taken from another direction, the top plate cut-out regions 1013 and 1014 with different inclusion methods and angles of the peripheral part, and the enlarged head cut-out region 1015 are arranged in the display image data. It is also possible to duplicate the same image data, change the magnification and rotation angle, and arrange them, or to stack and arrange a plurality of image data. However, the image data (top plate cut-out region and head cut-out region) used for confirming the subject 120 is arranged by rotating the image data so that the head-to-foot direction of the subject 120 approaches the vertical direction of the subject confirmation screen 1000. That is, the top plate cut-out region and the head cut-out region are arranged in a direction corresponding to the upright posture. Although the difference between the head-to-foot direction and the vertical direction of the subject confirmation screen 1000 increases, the image data may be rotated only at a predetermined rotation angle so as to reduce the computational load of the rotation process. For example, the image data may be rotated by 90 degrees, 180 degrees, or 270 degrees so that the difference between the head-to-foot direction and the vertical direction of the subject confirmation screen 1000 is reduced.

[0116] That is, the image processing device 102 identifies an object such as a part of the subject 120 or the device configuration of the radiation diagnostic device 104 in the camera image data, and generates rotated image data obtained by rotating the camera image data so that the identified object is in a predetermined direction. Then, the tablet PC 103 causes the display 140 to display the display image data including the rotated image data.

[0117] For example, when the object is the head of the subject 120, the image processing apparatus 102 generates rotated image data in which the head of the subject 120 is rotated so as to face upward, as shown in the head cutout area 1015 of FIG. 12. That is, when the object is the head of the subject 120, the image processing apparatus 102 generates rotated image data in which the head is upright.

[0118] Also, for example, when the object is the torso of the subject 120, the image processing apparatus 102 generates rotated image data in which the torso of the subject 120 is rotated so as to face upward. That is, when the object is the torso of the subject 120, the image processing apparatus 102 generates rotated image data in which the torso is upright.

[0119] Also, for example, when the object is a placement area such as the top plate 110 or the bed 111, the image processing apparatus 102 generates rotated image data in which the major axis of the placement area is vertical. For example, as shown in the top plate cutout area 1013 of FIG. 12, rotated image data is generated such that the center line direction of the top plate 110 is vertical.

[0120] The display image data shown in FIG. 12 is merely an example, and various modifications are possible. For example, as shown in FIG. 13, only the camera image data 1011 and the top plate cutout area 1013 rotated so that the top of the head faces upward may be arranged in the display image data. By arranging various images in the display image data, it becomes possible to present image data at an angle and within a range desired by the operator. On the other hand, by reducing the number of images arranged in the display image data, each image can be displayed larger and more clearly.

[0121] The full-screen display instruction button 1003 is a button for inputting an instruction to expand the image data drawing area 1001 to the full screen. The transition button 1004 to the layout setting screen is a button for inputting an instruction to transition from the subject confirmation screen 1000 to the layout setting screen described later. The display control function of the tablet PC 103 appropriately switches the display of the subject confirmation screen 1000 according to the input operation received via the full-screen display instruction button 1003 or the transition button 1004.

[0122] Next, the layout setting screen will be described. FIG. 14 is a diagram showing an example of the layout setting screen according to the fourth embodiment.

[0123] The layout setting screen 1100 includes, for example, a layout tab 1101, a composite layout tab 1102, a text box 1105 for inputting a layout name, a layout information registration button 1106, and a transition button 1107 to the subject confirmation screen. By selecting between the layout tab 1101 and the composite layout tab 1102, the arrangement of user interface components (hereinafter referred to as UI components) such as buttons and list boxes is changed.

[0124] On the screen when the layout tab 1101 is selected, for example, a coordinate system selection list box 1110, a camera number selection spin control 1111, a camera window 1112, a layout window 1113, slider controls 1120 to 1127, an attachment instruction button 1130, a paste area number selection spin control 1131, and a delete button 1132 are included. The coordinate system selection list box 1110 and the spin control 1111 are used for selecting the coordinate system and camera number of the image data to be displayed in the camera window 1112.

[0125] When "camera coordinate system" is selected in the coordinate system selection list box 1110, camera image data is displayed in the camera window 1112 such that the central pixel coincides with the center of the camera window 1112. The image data displayed in the camera window 1112 can be adjusted for the amount of rotation, magnification, horizontal shift distance, and vertical shift distance respectively using the slider controls 1120 to 1123. The scale intervals of the slider controls 1120 to 1123 can be arbitrarily given. However, since the rotation of the image data involves a large computational load, it may be discretized at 90-degree intervals, for example, 0 degrees, 90 degrees, 180 degrees, 270 degrees.

[0126] In the camera window 1112, a rectangle 1114 indicating the cutout area is overlaid and displayed on the image data. Since the rectangle 1114 indicating the cutout area is independent of the image data, it is not affected by the geometric transformation by the slider controls 1120 to 1123. The width and height of the rectangle 1114 indicating the cutout area can be adjusted using the slider controls 1124 and 1125.

[0127] The image data contained inside the rectangle 1114 indicating the cutout area is displayed on the layout window 1113 like the paste area 1115 shown in FIG. 14 after pressing the paste instruction button 1130. The position of the paste area 1115 within the layout window 1113 can be adjusted using the slider controls 1126 and 1127.

[0128] When the paste instruction button 1130 is pressed, the data input by the list box 1110, the spin control 1111, the slider controls 1120 to 1127, etc. is recorded in the memory of the tablet PC 103 together with the paste area number. The set of these recorded data is the layout information. By selecting the paste area number using the spin control 1131 and pressing the delete button 1132, the data associated with the paste area number can also be deleted from the record.

[0129] Next, a case will be described in which, in the coordinate system selection list box 1110, a coordinate system based on the features of the imaging target (subject 120, a part of the subject 120, top plate 110, bed 111, etc.) is selected instead of the camera coordinate system.

[0130] The coordinate system selection list box 1110 is an example of a user interface for specifying an object and a predetermined direction. The tablet PC 103 can receive an input of a part of the subject (such as the head or torso) via the coordinate system selection list box 1110. That is, the tablet PC 103 can specify an object and a predetermined direction by receiving an input of a part of the subject via the user interface. Also, the tablet PC 103 can receive an input of information regarding the placement area (such as the top plate 110 or the bed 111) via the coordinate system selection list box 1110. That is, the tablet PC 103 can specify an object and a predetermined direction by receiving an input of information regarding the placement area via the user interface.

[0131] FIG. 15 is a diagram showing an example of a case where a head coordinate system, which is one of the coordinate systems based on the features of the imaging target, is selected. The head coordinate system is the coordinate system of the head cutout area obtained by the head clipping step S209, with the center of the head as the origin and the vertex direction as the y-axis (the vertical coordinate axis of the head cutout area). However, the definition of this head coordinate system is not strict. For example, among the vertical and horizontal axes of the camera image, the direction closer to the vertex direction may be defined as the y-axis of the head coordinate system. At this time, by limiting the rotation angles that can be selected by the slider control 1120 to 0 degrees, 90 degrees, 180 degrees, and 270 degrees, the computational load of the image rotation process can be reduced. Also, a direction close to the vertex direction, such as the center line direction of the top plate, may be used instead.

[0132] When the "head coordinate system" is selected by the coordinate system selection list box 1110, a camera image rotated so that the head is upright is displayed in the camera window 1112. Similar to the case of other coordinate systems, after the paste instruction button 1130 is input, the image included in the rectangle 1150 of the cutout area is copied to the paste area 1151 according to the adjustment by the slider controls 1120 to 1127.

[0133] When the "head coordinate system" is selected, a camera selection condition check box 1152 is added to the screen. Since the head coordinate system is a coordinate system determined depending on the imaging object, if the imaging object is included in the image data, it can be defined regardless of the camera number. When the camera selection condition check box 1152 is checked, the camera number selected by the spin control 1111 is ignored, and the camera number with the largest head size obtained by the head detection step S207 is selected. Depending on the posture of the top plate, the head may be imaged small, but with this function, the camera image with the largest head among all cameras is selected as the paste area 1151. When the paste instruction button 1130 is pressed, the input value by the camera selection condition check box 1152 is also recorded as layout information together with the input data by other UI components.

[0134] After determining all the paste areas, by pressing the layout information registration button 1106, the layout information is registered in the image processing apparatus 102 together with the label input in the text box 1105. When this label is selected in the list box 1002 of the subject confirmation screen 1000, display image data is generated in consideration of the head coordinate system and the size of the head.

[0135] Next, the case where the composite layout tab 1102 is selected will be described. FIG. 16 is a diagram showing an example of a screen when the composite layout tab 1102 is selected. The composite layout switches a plurality of already registered layouts according to rules (hereinafter referred to as layout selection rules) using the characteristics of the shooting target. The screen when the composite layout tab 1102 is selected includes a rule selection list box 1160, a rule addition button 1161, a rule number spin control 1162, a rule deletion button 1163, and a rule detail input section 1164.

[0136] The composite layout shown in FIG. 16 is an example of a user interface for specifying an object and a predetermined direction. The tablet PC 103 can receive an input of conditions (layout selection rules) regarding the object included in the camera image data according to the composite layout. That is, the tablet PC 103 can specify the object and a predetermined direction by receiving an input of conditions regarding the object through the user interface.

[0137] UI components are arranged in the rule detail input section 1164 according to the layout selection rule selected in the rule selection list box 1160. When the "center line direction switching" rule is selected in the rule selection list box 1160, the rule detail input section 1164 includes a camera number input spin control 1170, a top plate center line angle input text box 1171, a top plate center line angle condition selection list box 1172, and a layout selection list box 1173.

[0138] The information obtained from the UI components of the rule details input section 1164 represents the layout selection rules. Let the number input by the spin control 1170 be "A", the angle input in the text box 1171 be "B", the angle condition of the list box 1172 be "C", and the layout of the layout selection list box 1173 be "D". In this case, the layout selection rule is "Obtain the angle of the top plate center line in the A-th camera image data, compare it with B, and select layout D when the comparison result satisfies the angle condition C".

[0139] After inputting data in the rule details input section 1164, by pressing the rule addition button 1161, the layout selection rule is recorded in the memory provided in the tablet PC 103. The number of the layout selection rule can be selected by the spin control 1162, and the record of the selected layout rule can be deleted by the rule deletion button 1163.

[0140] In the case of a composite layout, the set of recorded layout selection rules is treated as layout information. By pressing the layout information registration button 1106, the layout information is registered in the image processing apparatus 102 together with the label input in the text box 1105.

[0141] As described above, the image processing apparatus 102 according to the fourth embodiment identifies an object in the camera image data and generates rotated image data obtained by rotating the camera image data so that the identified object is in a predetermined direction. Further, the tablet PC 103 causes the display 140 to display the display image data including the rotated image data. With such a configuration, the monitoring camera system 100 according to the fourth embodiment can assist in the efficient monitoring of the subject 120. That is, the monitoring camera system 100 can rotate and display the image data in a direction determined based on the characteristics of the tracking target such as the head or the top plate, and assist the operator in monitoring the tracking target.

[0142] In each of the above-described embodiments, the radiation diagnostic apparatus 104 has been described as an example of a medical device on which a subject can be placed. However, the embodiments are not limited thereto, and the present invention can be similarly applied to other medical devices having a top plate, a bed, or the like and on which a subject can be placed. Examples of such other medical devices include medical imaging diagnostic devices such as X-ray CT devices and MRI devices, and treatment devices such as radiation treatment devices. In the case of a device equipped with a gantry such as an X-ray CT device or an MRI device, the camera 30 may be provided on the gantry.

[0143] The term "processor" used in the above description means, for example, a circuit such as a CPU, a GPU (Graphics PROCESSING Unit), an ASIC, or a programmable logic device (for example, a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)). When the processor is, for example, a CPU, the processor realizes its function by reading and executing a program stored in a storage circuit. On the other hand, when the processor is, for example, an ASIC, instead of storing a program in a storage circuit, the function is directly incorporated as a logic circuit in the circuit of the processor. Note that each processor of the embodiments is not limited to being configured as a single circuit for each processor, and a plurality of independent circuits may be combined to be configured as one processor to realize its function. Further, a plurality of components in each figure may be integrated into one processor to realize its function.

[0144] Each component of each device according to the above-described embodiment is a functional concept and does not necessarily have to be physically configured as shown in the drawings. That is, the specific form of the distribution and integration of each device is not limited to that shown in the drawings, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads and usage situations. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware by wired logic.

[0145] Also, the method described in the above-described embodiment can be realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. This program can be distributed via a network such as the Internet. Further, this medical information processing program is recorded on a non-transitory recording medium readable by a computer such as a hard disk, a flexible disk (FD), a CD-ROM, an MO, a DVD, etc., and can also be executed by being read from the recording medium by a computer.

[0146] According to at least one of the embodiments described above, efficient monitoring of a subject can be supported.

[0147] Regarding the above embodiments, the following additional remarks are disclosed as one aspect and selective features of the invention. (Additional Remark 1) A camera that photographs an environment including a medical device installed in an examination room and capable of placing a subject, An image processing unit that generates display image data based on the camera image data obtained by the camera, A display control unit that causes the display unit to display the display image data A monitoring camera system comprising: The image processing unit: Estimates a placement area corresponding to the area where the subject is placed in the camera image data, Generate corrected image data obtained by correcting camera image data according to the placement area, Estimate the area of the subject included in the corrected image data, A surveillance camera system that generates display image data based on the corrected image data and the area of the subject. (Appendix 2) The placement area may be the top plate of the medical device. (Appendix 3) The placement area may be the bed of the medical device. (Appendix 4) The placement area may be estimated by a deep neural network. (Appendix 5) The placement area may be estimated based on the detection result of a marker attached to the top plate or the bed of the medical device and the position and orientation of the camera. (Appendix 6) The placement area may be estimated based on the control information of the top plate or the bed of the medical device and the position and orientation of the camera. (Appendix 7) The image processing unit may generate the corrected image data by rotating the area including the placement area so that the center line of the placement area is in the vertical direction. (Appendix 8) The image processing unit may generate the corrected image data by converting the area including the placement area so as to approach a vertically long rectangle. (Appendix 9) The rotation angle of the rotation may be determined based on the direction of the side of the polygon surrounding the placement area. (Appendix 10) The rotation angle of the rotation may be determined based on the difference in the positions of objects commonly included in a plurality of the camera image data taken at different times. (Appendix 11) A camera installed in an examination room and photographing an environment including a medical device on which a subject can be placed, An image processing unit that generates display image data based on the camera image data obtained by the camera, A display control unit that causes the display unit to display the display image data A surveillance camera system comprising The image processing unit Identifies an object in the camera image data Generates rotated image data by rotating the camera image data so that the identified object is in a predetermined direction The display control unit causes the display unit to display the display image data including the rotated image data. A surveillance camera system (Appendix 12) The object may be the head of the subject, and the predetermined direction may be the upward direction (Appendix 13) The object may be the torso of the subject, and the predetermined direction may be the upward direction (Appendix 14) The object may be a placement area corresponding to the area where the subject is placed, and the predetermined direction may be the direction in which the major axis of the placement area is vertical (Appendix 15) The display control unit may cause the display unit to display a user interface for designating the object and the predetermined direction (Appendix 16) The display control unit may specify the object and the predetermined direction by receiving an input of the inspection content of the medical device via the user interface (Appendix 17) The display control unit may specify the object and the predetermined direction by receiving an input of user information of the medical device via the user interface (Appendix 18) The display control unit may specify the object and the predetermined direction by receiving an input of the part of the subject via the user interface (Appendix 19) The display control unit may specify the object and the predetermined direction by receiving an input of information regarding a placement area corresponding to the area on which the subject is placed via the user interface. (Appendix 20) The display control unit may specify the object and the predetermined direction by receiving an input of conditions regarding the object included in the camera image data via the user interface. (Appendix 21) A radiation diagnostic apparatus capable of placing a subject, A camera installed in an examination room and photographing an environment including the radiation diagnostic apparatus, An image processing unit that generates display image data based on the camera image data obtained by the camera, A display control unit that causes the display unit to display the display image data A radiation imaging system comprising: The image processing unit, estimates a placement area corresponding to the area on which the subject is placed in the camera image data, generates corrected image data obtained by correcting the camera image data according to the placement area, estimates the area of the subject included in the corrected image data, A radiation imaging system that generates display image data based on the corrected image data and the area of the subject. (Appendix 22) A radiation diagnostic apparatus capable of placing a subject, A camera installed in an examination room and photographing an environment including the radiation diagnostic apparatus, An image processing unit that generates display image data based on the camera image data obtained by the camera, A display control unit that causes the display unit to display the display image data, A radiation imaging system comprising: The image processing unit, identifies an object in the camera image data, Generate rotated image data by rotating the camera image data so that a specific object is in a predetermined direction. The display control unit causes the display unit to display the display image data including the rotated image data. A radiation imaging system. (Appendix 23) Based on camera image data obtained by a camera that photographs an environment including a medical device installed in an examination room and on which a subject can be placed, generate display image data. A method including causing a display unit to display the display image data, Estimate a placement area corresponding to the area where the subject is placed in the camera image data. Generate corrected image data by correcting the camera image data according to the placement area. Estimate the area of the subject included in the corrected image data. Generate display image data based on the corrected image data and the area of the subject. A method. (Appendix 24) Based on camera image data obtained by a camera that photographs an environment including a medical device installed in an examination room and on which a subject can be placed, generate display image data. A method including causing a display unit to display the display image data, Identify an object in the camera image data. Generate rotated image data by rotating the camera image data so that the identified object is in a predetermined direction. The display control unit causes the display unit to display the display image data including the rotated image data. A method.

[0148] Although several embodiments have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, changes, and combinations of the embodiments can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and its equivalent scope.

Explanation of Symbols

[0149] 1: System 10: Medical device 20: Information processing device 21: Communication interface 22: Input interface 23: Display 24: Memory 25: Processing circuit 25a: Control function 25b: Image processing function 25c: Display control function 30: Camera 100: Surveillance camera system 101: IP camera 102: Image processing device 103: Tablet PC 104: Radiological diagnostic device 105: Wireless network hub 106: Network 107: Network cable 110: Ceiling 111: Bed 112: Radiation (X-ray) tube 113: Column 120: Subject 130: X-axis of the ceiling coordinate system 131: Y-axis of the ceiling coordinate system 132: Z-axis of the ceiling coordinate system 140: Display 301: Ceiling 302: Head 303: Torso 304: Foot 305: Side 306: Side 307: Center line 308: Center of gravity 321: Region 322: Ceiling edge polygon 330: Ceiling cut-out region 600: Surveillance camera system 601: IP camera 602: IP camera 603: Image processing device 604: Display 605: Wired network hub 606: Display cable 610: Marker 611: Marker 640: Display screen 801: Corner point 802: Marker 810: x-axis 811: y-axis 812: z-axis 813: Origin 1000: Subject confirmation screen 1001: Image data drawing area 1002: List box for layout selection 1003: Full screen display instruction button 1004: Transition button 1011: Camera image data 1012: Camera image data 1013: Ceiling cutout area 1014: Ceiling cutout area 1015: Head cutout area 1100: Layout setting screen 1101: Layout tab 1102: Composite layout tab 1105: Text box for layout name input 1106: Layout information registration button 1107: Transition button 1110: List box for coordinate system selection 1111: Spin control for camera number selection 1112: Camera window 1113: Layout window 1114: Rectangle 1115: Pasting area 1120: Slider control 1121: Slider control 1122: Slider control 1123: Slider control 1124: Slider Control 1125: Slider Control 1126: Slider Control 1127: Slider Control 1130: Attachment Instruction Button 1131: Spin Control for Selecting Attachment Area Number 1132: Delete Button 1150: Rectangle 1151: Attachment Area 1152: Camera Selection Condition Check Box 1160: Rule Selection List Box 1161: Rule Addition Button 1162: Spin Control for Rule Number 1163: Rule Delete Button 1164: Rule Details Input Section 1170: Spin Control for Entering Camera Number 1171: Text Box for Entering the Angle of the Top Plate Center Line 1172: List Box for Selecting the Condition of the Top Plate Center Line Angle 1173: List Box for Selecting Layout

Claims

1. A camera installed in an examination room for photographing an environment including a medical device on which a subject can be placed, an image processing unit that generates display image data based on the camera image data obtained by the camera, a display control unit that causes the display unit to display the display image data, A monitoring camera system comprising: wherein the image processing unit estimates a placement area corresponding to an area where the subject is placed in the camera image data, generates corrected image data obtained by correcting the camera image data according to the placement area, estimates an area of the subject included in the corrected image data, A monitoring camera system that generates display image data based on the corrected image data and the area of the subject.

2. The monitoring camera system according to claim 1, wherein the placement area is the top plate of the medical device.

3. The monitoring camera system according to claim 1, wherein the placement area is the bed of the medical device.

4. The monitoring camera system according to claim 1, wherein the placement area is estimated by a deep neural network.

5. The monitoring camera system according to claim 1, wherein the placement area is estimated based on a detection result of a marker attached to the top plate or the bed of the medical device and the position and orientation of the camera.

6. The monitoring camera system according to claim 1, wherein the placement area is estimated based on control information of the top plate or the bed of the medical device and the position and orientation of the camera.

7. The monitoring camera system according to claim 1, wherein the image processing unit generates the corrected image data by rotating an area including the placement area so that the center line of the placement area is in the vertical direction.

8. The monitoring camera system according to claim 1, wherein the image processing unit generates the corrected image data by converting an area including the placement area so as to approach a vertically long rectangle.

9. The monitoring camera system according to claim 7, wherein the rotation angle of the rotation is determined based on the direction of a side of a polygon surrounding the placement area.

10. The monitoring camera system according to claim 7, wherein the rotation angle of the rotation is determined based on a difference in positions of objects commonly included in a plurality of the camera image data at different shooting times.

11. A camera installed in an examination room for photographing an environment including a medical device on which a subject can be placed, an image processing unit that generates display image data based on the camera image data obtained by the camera, A display control unit that causes the display unit to display the display image data; A surveillance camera system comprising: The image processing unit: Identifies an object in the camera image data, Generates rotated image data obtained by rotating the camera image data so that the identified object faces a predetermined direction, The display control unit causes the display unit to display the display image data including the rotated image data. A surveillance camera system. **Claim 12** The surveillance camera system according to claim 11, wherein the object is the head of the subject, and the predetermined direction is the upward direction. **Claim 13** The surveillance camera system according to claim 11, wherein the object is the torso of the subject, and the predetermined direction is the upward direction. **Claim 14** The surveillance camera system according to claim 11, wherein the object is a placement area corresponding to an area where the subject is placed, and the predetermined direction is a direction in which the major axis of the placement area is vertical. **Claim 15** The surveillance camera system according to claim 11, wherein the display control unit causes the display unit to display a user interface for designating the object and the predetermined direction. **Claim 16** The surveillance camera system according to claim 15, wherein the display control unit designates the object and the predetermined direction by receiving an input of inspection content of the medical device via the user interface. **Claim 17** The surveillance camera system according to claim 15, wherein the display control unit designates the object and the predetermined direction by receiving an input of user information of the medical device via the user interface. **Claim 18** The surveillance camera system according to claim 15, wherein the display control unit designates the object and the predetermined direction by receiving an input of a part of the subject via the user interface. **Claim 19** The surveillance camera system according to claim 15, wherein the display control unit designates the object and the predetermined direction by receiving an input of information regarding a placement area corresponding to an area where the subject is placed via the user interface. **Claim 20** The monitoring camera system according to claim 15, wherein the display control unit designates the object and the predetermined direction by receiving an input of conditions regarding the object included in the camera image data via the user interface.

21. A radiation diagnostic apparatus on which a subject can be placed, A camera installed in an examination room and photographing an environment including the radiation diagnostic apparatus, An image processing unit that generates display image data based on the camera image data obtained by the camera, A display control unit that causes the display unit to display the display image data A radiation imaging system comprising: The image processing unit: Estimates a placement area corresponding to an area where the subject is placed in the camera image data, Generates corrected image data obtained by correcting the camera image data according to the placement area, Estimates the area of the subject included in the corrected image data, A radiation imaging system that generates display image data based on the corrected image data and the area of the subject.

22. A radiation diagnostic apparatus on which a subject can be placed, A camera installed in an examination room and photographing an environment including the radiation diagnostic apparatus, An image processing unit that generates display image data based on the camera image data obtained by the camera, A display control unit that causes the display unit to display the display image data, A radiation imaging system comprising: The image processing unit: Identifies an object in the camera image data, Generates rotated image data obtained by rotating the camera image data so that the identified object is in a predetermined direction, A radiation imaging system, wherein the display control unit causes the display unit to display the display image data including the rotated image data.

23. A method of generating display image data based on camera image data obtained by a camera installed in an examination room and photographing an environment including a medical device on which a subject can be placed, and causing the display unit to display the display image data, the method comprising: Estimating a placement area corresponding to an area where the subject is placed in the camera image data, Generating corrected image data obtained by correcting the camera image data according to the placement area, Estimating the area of the subject included in the corrected image data, Generating display image data based on the corrected image data and the area of the subject.

24. ​ Based on camera image data obtained by a camera that photographs an environment including a medical device installed in an examination room and on which a subject can be placed, display image data is generated, A method including displaying the display image data on a display unit, identifying an object in the camera image data, generating rotated image data obtained by rotating the camera image data so that the identified object is in a predetermined direction, The display control unit causes the display unit to display the display image data including the rotated image data.

Images