Subject Motion Measurement Device, Subject Motion Measurement Method, Program, Imaging System

Abstract

To provide a technique capable of accurately measuring the movement of a subject.SOLUTION: A subject motion measuring device includes a camera for photographing a subject and a processing unit for acquiring three-dimensional coordinate data of a patch set in the frame data included in the moving image data photographed by the camera. The processing unit tracks the patch in the moving image data photographed by the camera, and calculates the motion data of the subject on the basis of the three-dimensional coordinate data of the tracked patch.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a technique for measuring the movement of a subject.

Background Art

[0002] In recent years, magnetic resonance imaging (MRI) devices with a static magnetic field strength of 3 Tesla have been commercialized, and the resolution of output images has been improving. However, artifacts that were relatively inconspicuous in the past have become more prominent, and improvement is desired as a new issue. One of the causes of such artifact generation is known to be the movement of the head within the head coil, and attempts have been made to correct the gradient magnetic field of the MRI device according to the head movement measured by a camera.

[0003] Patent Document 1 discloses a method of detecting head movement by attaching a marker called RGR (Retro-grate reflector) to the face, photographing it with a camera from outside the head coil, and detecting the difference in marker position between video frames. RGR is a marker that uses moiré fringes for position detection and enables highly accurate position measurement.

[0004] Markers are important for improving detection accuracy, but markerless methods are also being studied in order to avoid the trouble of attaching and detaching the subject's face and the irritation to the skin at the attachment location. Non-Patent Document 1 discloses a method of detecting head movement based on the difference in three-dimensional shape between different video frames by measuring the three-dimensional shape of the surface near the nose using pattern illumination.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Documents

[0006]

Non-Patent Document 1

[0007] An object of the present invention is to provide a technique capable of accurately measuring the movement of a subject by a markerless method. Means for Solving the Problems

[0008] The subject movement measuring apparatus of the present invention is a processing unit that acquires three-dimensional coordinate data of a patch set for image data including a predetermined part of the subject and a peripheral part of the predetermined part, which are acquired from an opening of the head coil in frame data included in moving image data obtained by photographing the subject wearing a head coil with a camera and an estimation unit that estimates the position of the patch based on reference coordinate data serving as a reference for the subject and the processing unit tracks the patch in the moving image data photographed by the camera and calculates movement data of the head of the subject based on the three-dimensional coordinate data of the tracked patch and when the patch moves outside the frame of the frame data, the estimation unit estimates the position of the patch . Effects of the Invention

[0009] According to the present invention, the movement of a subject can be accurately measured. Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Embodiments for Carrying Out the Invention

[0011] An overview of a subject movement measurement method, a subject movement measurement device, and an imaging system according to an embodiment of the present invention will be described. However, the present invention is not limited thereto. [Subject Movement Measurement Method] Hereinafter, a subject movement measurement method according to an embodiment of the present invention will be described with reference to the drawings.

[0012] [Terms Used in the Description of the Subject Movement Measurement Method] In this specification, a first imaging means for optically photographing a subject for measuring the movement of the subject and a second imaging means for imaging the inside of the subject appear. The former is, for example, a camera, and the latter is a modality such as a magnetic resonance imaging device. In order to distinguish between the two, in this specification, terms such as "camera", "imaging means", and "imaging" are used in the description of the former, and terms such as "modality", "imaging device", "imaging means", and "imaging" are used in the description of the latter.

[0013] The "subject" is an object to be measured for movement, that is, an object photographed by a camera which is the first imaging means. Typically, a part of the subject (living body) corresponds to the subject. In the embodiments described below, the "peripheral part of the eye" of the human head is shown as an example of the subject, but it is not limited thereto, and the head, face, a part of the face (peripheral part of the facial organ), chest, abdomen, upper limb, lower limb, or other part of the body may be used as the subject. This measurement method treats the overall movement of the subject (referred to as "overall movement") and the local movement on the surface of the subject (referred to as "local movement") as equivalent. The local movement can be described as a change in the position and posture of a point or a minute region on the subject in a coordinate system. For example, when the "peripheral part of the eye" is the subject, the change in the position and posture of the peripheral part of the eye due to the head movement (translation and rotation of the entire head) corresponds to the overall movement, and the movement of the skin around the organs such as the eyes and eyebrows corresponds to the local movement.

[0014] The moving image data handled by the subject movement measurement method will be described. The moving image data is data obtained by photographing the peripheral part of the eyes, which are the subject, from different directions using a plurality of cameras. The photographing is performed using two or more cameras for one eye, or two or more cameras for each of the two eyes. For example, when there are two cameras for each eye, the camera arrangement as seen from above the head is as shown in Fig. 1A, and the camera arrangement as seen from the lateral direction of the head is as shown in Fig. 1B. In addition to the method using two or more cameras (the so-called passive stereo method), the amount of calculation for obtaining the parallax can be reduced by the method of projecting a pattern onto the subject using a projector (the so-called active stereo method). However, Fig. 1A shows an example of an arrangement using only cameras. The first image data and the second image data photographed by a plurality of cameras are image data photographed from different positions in the plurality of cameras. The first image data and the second image data are image data photographed from different directions of the peripheral part of a predetermined part of the subject. The same region of the subject is photographed in the first image data and the second image data.

[0015] The frame rate of the moving image data can be arbitrarily selected. However, in order to smoothly measure the movement of the head (skin), 50 fps or more is desirable. Although it is desirable that the number of pixels of the image data (hereinafter referred to as a video frame) included in the moving image data is large, it is limited by the frame rate. The number of pixels is, for example, about 1 megapixel (about 1300×800 pixels).

[0016] The shooting magnification of the camera is selected so that the peripheral part of the video frame (the part away from the image center) includes textures (such as eyebrows and skin patterns). The texture is used as a mark for the tracking process, but there are also textures that are not suitable for the tracking process, such as parts with fast movement and large deformation (eyes and lower eyelids). Therefore, as shown in Fig. 1C, it is advisable to set the range of the video frame to the region 110 including the eyes at the center or the region 111 not including the eyes.

[0017] The video frames of two cameras that capture a subject from different angles contain a subject with a displacement. The above displacement is called parallax and occurs according to the difference in height (depth map) of the subject surface.

[0018] In the tracking process, three-dimensional coordinate data of a pixel section (hereinafter referred to as a patch or a small area) including the texture is calculated using parallax (Figure 2). In the calculation of the parallax 131, an image 126 (hereinafter referred to as reference plane image data 127) obtained by projecting the video frame onto a plane common to each camera 121 (hereinafter referred to as the reference plane 125) is created, and the difference in the positions of patches 128 commonly included in a plurality of reference plane image data 127 is obtained. The reference plane 125 is a plane approximating the subject or a plane parallel to the above plane. The intersection of the principal ray 123 passing through the aperture center 120 of one camera and the position of the patch 129 on the reference plane 125 and the principal ray 123 passing through the aperture center 120 of the other camera and the position of the patch 129 on the reference plane 125 is the three-dimensional coordinates of the patch 130 on the subject. The direction in which the parallax 131 occurs can be adjusted according to the direction of the optical axis 124 of each camera and the coordinates of the aperture center 120. It is preferable to adjust so that the direction in which the parallax 131 occurs is parallel to the coordinate axis (horizontal axis) of the video frame to facilitate the calculation of the three-dimensional coordinate data of the patch.

[0019] Note that the coordinate axes of the three-dimensional coordinate data include the x and y axes within the reference plane 125, and the z axis is parallel to the normal line of the reference plane 125. When selecting and tracking a patch, two-dimensional coordinate data within the reference plane image data is handled, and the coordinate axes of the two-dimensional coordinate data are the same as the x and y axes of the three-dimensional coordinate data. Since the calculation of the two-dimensional coordinate data mainly uses pixels as units, the units are often different from those of the calculation of the three-dimensional coordinate data. In the description of the calculation of the three-dimensional coordinate data based on the two-dimensional coordinate data, it shall be adjusted to the unit of the three-dimensional coordinate data without special notice.

[0020] <Regarding the overall flow of the subject movement measurement method> The process of the subject movement measurement method according to this embodiment will be described. In the frame data included in the moving image data captured by the camera, the three-dimensional coordinate data of the patch (small area) set is acquired, and the patch is tracked in the moving image data captured by the camera. Then, based on the three-dimensional coordinate data of the tracked patch, the movement data of the subject is calculated. Specifically, based on the parallax between the first image data and the second image data in the moving image data captured by a plurality of cameras, the three-dimensional coordinate data of the patches set in the first image data and the second image data is acquired. From the three-dimensional coordinate data of the patch and the reference coordinate data serving as the reference of the subject, the translation and rotation of the subject are calculated. The patches included in the first image data and the second image data are of the same size and the same shape respectively. Based on the parallax between the first image data and the second image data, the three-dimensional coordinate data of the patches set in the first image data and the second image data is acquired. From the three-dimensional coordinate data of the patch and the reference coordinate data serving as the reference of the subject, the translation and rotation of the subject are calculated. The patches included in the first image data and the second image data are of the same size and the same shape respectively.

[0021] The patch is a small area set in the band area in the frame data. The patch is set in the band area around a predetermined part of the subject. In this embodiment, a plurality of patches are set, and each patch is separated from each other by a predetermined distance or more and is set in an area where the movement of the skin does not affect each other. Here, the predetermined part is a part that can be photographed by a plurality of cameras from the gap of the receiving coil. As an example, the predetermined part is the eye or nose of the subject.

[0022] Next, an example of the procedure of the tracking process will be specifically described with reference to FIG. 3A. In the calibration process S510, before acquiring the moving image, the imaging system parameters (the direction vector of the optical axis for each camera and the aperture center coordinates) are acquired. The imaging system parameters are calculated based on the design values or measured values of the camera optical system, or a calibration chart including landmarks is photographed, and calculated based on the positions of the landmarks on the video frames. The calibration process S510 does not need to be executed every time when measuring the head movement, and can be omitted if it is obvious that there is no variation in the imaging system parameters.

[0023] After the start of shooting a moving image, in the patch selection step S511, for a plurality of cameras that are shooting the same subject, a plurality of patches to be tracked are selected based on video frames with close shooting times. In the patch tracking step S512, two-dimensional coordinate data of the destination where the patch selected in the patch selection step S511 has moved due to the movement of the subject / skin is measured from within the video frame based on the texture included in the patch. The two-dimensional coordinate data is recorded in an array. In the 3D tracking step S513, for each shooting time of the video frame, three-dimensional coordinate data of the patch is calculated based on the two-dimensional coordinate data of the patch obtained for each camera, and is recorded in an array.

[0024] In the reference coordinate holding determination S514, it is determined whether the three-dimensional coordinate data of the patch (hereinafter referred to as reference coordinate data) that is the reference (translation 0 mm, rotation 0 deg) for the head movement is being held. If it is not being held, the reference coordinate data is recorded in the array in the reference coordinate recording step S515. The three-dimensional coordinate data of the patch obtained at the current time may be adopted as the reference coordinate data. If the reference coordinate data is being held, the subject movement calculation step S516 is carried out. In the subject movement calculation step S516, subject movement data is calculated from the three-dimensional coordinate data of the patch obtained at the current time and the three-dimensional coordinate values included in the reference coordinate data.

[0025] In the failed patch estimation step S517, the two-dimensional coordinates of the patches that have moved outside the video frame and the patches that have failed in the calculation in the patch tracking step S512 are estimated and added to the two-dimensional coordinate data recorded in the patch tracking step S512. The estimated two-dimensional coordinate data is used when executing the patch tracking step S512 in the video frame at the next shooting time. In the failed patch estimation step S517, when the patch (first patch) set in the frame data of the moving image has moved outside the frame, the position of the patch (first patch) is estimated based on the reference coordinate data serving as the reference of the subject. In the failed patch estimation step S517, based on the patch (second patch) within the frame among the plurality of patches, the position of the patch (first patch) that has moved outside the frame may be estimated. Specifically, the position of the patch that has moved outside the frame within the frame is estimated based on the center coordinates of the plurality of patches. Then, when the patch (first patch) enters the frame, the tracking of the patch (first patch) is immediately resumed.

[0026] In the skin movement data acquisition determination S518, it is determined whether a creation completion notification has been sent. If there is no creation completion notification, return to the patch tracking step S512 for the video frame at the next shooting time.

[0027] <Regarding the patch selection step of the tracking step> An example of the procedure of the patch selection step S511 will be described with reference to FIG. 4A. In the acquisition of video frames S530, a plurality of initial video frames at the shooting time are acquired for each camera.

[0028] In the black eye detection step S531, the position and size of the black eyes in the plurality of video frames are detected, and a plurality of video frames in which the black eyes are close to the center of the video frame are selected. The black eyes can be detected by threshold processing on the pixel values. Note that when the eyes are not included in the video frame or when the head is fixed and the position of the eyes does not change within the video frame, the black eye detection step S531 can be omitted.

[0029] In the contrast detection step S532, the contrast of the skin texture included in the plurality of video frames obtained in the black eye detection step S531 (when omitted, the acquisition of video frames S530) is calculated. Next, video frames with high contrast (one video frame with the same shooting time for each camera) are selected. As a result, video frames with little motion blur can be selected. The contrast can be calculated by, for example, the Normalized variance given by the following formula.

Equation

[0030] In the selectable region extraction step S533, reference plane image data (hereinafter referred to as the selectable region) is created based on the pixels in the periphery of the video frame selected in the contrast detection step S532. As shown in FIG. 5A, it is desirable that the sizes of the upper and lower selectable regions be equal. When creating the reference plane image data, noise removal of the video frame may be performed simultaneously. For example, noise with a fixed pattern causes errors during parallax calculation and patch tracking described later, so it is removed using a median filter.

[0031] In the band region extraction step S534, a horizontally long region (hereinafter referred to as the band region) is cut out from the selectable region for each camera, and pixel value normalization is performed (FIG. 5B). Pixel value normalization is a process of making the reflected light intensity from the surface of the subject approach a value that does not depend on the optical axis direction of the camera. That is, the controller (processing unit) 1153 described later performs pixel value normalization in the band regions of the first image data and the second image data captured by a plurality of cameras. For example, when normalization is performed according to the following formula, an approximate value of the pigment concentration on the surface of the subject is obtained, and a value with low dependence on the camera optical axis direction can be obtained.

Equation

[0032] After the division of band region 2, a correlation operation between band region 1 and band region 2A and a correlation operation between band region 1 and band region 2B are performed, and the height and position of the peak are calculated from each correlation distribution. The correlation operation is given by the following equation.

[0033]

Equation

Equation

[0034] After calculating the peak height and peak position of the correlation distribution, select the band region with the higher peak (assume it is band region P). The average disparity, which is the output value of the average disparity calculation step S535, is the difference between the peak position of band region P and the center position of band region P within band region 2.

[0035] When there are three or more cameras (assume N cameras) simultaneously photographing the subject part, two cameras can be selected from the N cameras, and the procedure of performing the average disparity calculation step S535 in the case of two cameras can be repeated. However, depending on the camera arrangement, the direction in which the disparity occurs may be different from the long side direction of the band region. In that case, it is necessary to avoid the camera combination or calculate band regions with different directions in advance in the band region extraction step S534.

[0036] In the patch disparity calculation step S536, patch candidates are selected from within the band region, and the disparity for each patch candidate is calculated. Similar to the explanation of the average disparity calculation step S535, the case where there are two cameras simultaneously photographing the subject part will be described first. The band regions for the two cameras are distinguished and called band regions 1 and 2 (or the first image and the second image).

[0037] First, a region in which the same subject part is included in band region 1 and band region 2 is calculated based on the average disparity obtained in the average disparity calculation step S535. For example, when the relationship between band regions 1 and 2 and the average disparity is as shown on the left side of FIG. 6, a region obtained by deleting an offset equal to the average disparity from band region 2 (hereinafter referred to as band region 2C) includes the same subject part as band region 1.

[0038] After selecting band region 2C, a plurality of patch candidates (small region candidates) are selected from within band region 2C. If the size of the patch candidate is too large, it will lead to an increase in the calculation time of the patch tracking step S512 during head movement measurement. It is preferably a square having a size close to the length of the short side of the band region.

[0039] After obtaining the above patch candidates, a correlation operation (Equation 3) between band region 1 and each patch candidate is performed, and the height and position of the peak are calculated from each correlation distribution. Similar to the average disparity calculation step S535, Phase correlation or template matching may be used instead of the correlation operation. The difference between the above peak position and the patch candidate position within band region 2 is defined as the disparity for each patch candidate.

[0040] The disparity calculated for each of the above patch candidates may be a false value generated due to misrecognition of the texture pattern, and it is necessary to remove patch candidates for which the correct disparity cannot be obtained. The correctness of the disparity for each of the above patch candidates (hereinafter referred to as disparity reliability) is related to the normal line of the subject surface on which the patch candidate is located (hereinafter referred to as the patch candidate normal line). The disparity reliability will be described based on the relationship between the orientation of the patch candidate and the reduction rate of the patch candidate image included in the reference plane data (FIGS. 7A and 7B). In the calculation of the patch candidate image included in the reference plane data, two straight lines AO and BO passing through two points A and B corresponding to two sides of the patch candidate and the aperture center O are obtained, and the intersections C and D of the two straight lines AO and BO with the reference plane are obtained. The line segment CD corresponds to the patch candidate image included in the above reference plane data, and the reduction rate of the patch candidate image is the ratio of the length of the line segment CD to the length of the line segment AB (the size of the patch candidate).

[0041] When the direction of the patch candidate normal is the same as the direction of the reference plane normal (Figure 7A), the reduction ratio for each camera is equal, so the patch candidate images (line segments CD and C'D') calculated for each camera are the same. On the other hand, when the direction of the patch candidate normal is not the same as the direction of the reference plane normal (Figure 7B), a difference occurs in the reduction ratio, so the patch candidate images calculated for each camera do not match.

[0042] If the patch candidate images are different for each camera, the height of the correlation peak between the band region 1 and the patch candidate decreases, increasing the likelihood of erroneously detecting the second and third peaks. From the above, the parallax reliability decreases when the direction of the patch candidate normal is not the same as the direction of the reference plane normal.

[0043] In this embodiment, based on a plurality of peaks in the autocorrelation distribution of the patch candidate image included in the reference plane data and the average parallax obtained in the average parallax calculation step S535, a patch candidate with high parallax reliability is selected. The autocorrelation distribution is the correlation distribution when I1 of Equation 3 and I t are used as the pixel value distribution of the patch candidate image included in the reference plane data. Ideally, the peak height of the patch candidate with high parallax reliability is a value close to the peak height of the autocorrelation distribution. Also, if there is a second highest peak in the autocorrelation distribution and its height is close to that of the first peak, the second peak may be used for parallax detection, resulting in the occurrence of false parallax. Therefore, the ratio of the heights of the first and second peaks is also a criterion for parallax reliability.

[0044] The reason why the average parallax can be used as the second criterion will be explained. The surface shape near the eyes has a small height difference, and there are many locations on the surface of the subject that are parallel to the surface (i.e., the reference plane) approximated by a plane. As a result, when calculating the average parallax for the band region, it approaches the parallax of the locations parallel to the reference plane. Since the parallax reliability of the patch candidates parallel to the reference plane is high, the average parallax can be used as a reference.

[0045] In the selection of patch candidates with high parallax reliability, for example, three-stage screening can be performed. First, the first screening is performed so that the height of the correlation peak between the band region 1 and the patch candidate is 50% or more of the peak height of the autocorrelation distribution. In the second screening, those with the height of the second peak relative to the first peak of the autocorrelation distribution being less than 70% are selected. In the third screening, patch candidates close to the average parallax are selected. If a specific screening does not function, some of the three-stage processes may be omitted. Also, when using template matching instead of the correlation operation, a value such as the degree of template matching may be used instead of the autocorrelation peak. That is, a plurality of regions are set for each of the images of a plurality of frames acquired by a plurality of cameras, and based on the reliability of each patch, what is to be used as a patch is selected.

[0046] When there are three or more cameras (N cameras) simultaneously photographing the subject part, similar to the average parallax calculation step S535, two cameras are selected from the N cameras, and the procedure of implementing the patch parallax calculation step S536 for the case of two cameras may be repeated.

[0047] In the patch number determination S537, it is determined whether a predetermined number of patch candidates have been obtained. If not, the band regions at different positions are calculated by the band region extraction step S534 to increase the number of patches. In this way, a plurality of patches to be tracked can be selected by the patch selection step S511.

[0048] <Regarding the patch tracking process in the tracking process> An example of the procedure of the patch tracking step S512 will be described with reference to FIG. 4B. In the exploration area acquisition step S700, reference plane image data is created for the newly acquired video frame, and a pixel section (hereinafter referred to as the exploration area) that may include a patch is acquired. The center coordinates of the exploration area are estimated based on the center coordinate data of the patch (hereinafter referred to as 2D data) at the shooting time before the current time. For example, if the 2D data of the patch at the past shooting times t1 and t2 are x1 and x2, the center coordinates x3 of the exploration area at the current shooting time t3 can be calculated by the following formula.

Equation

[0049] In the 2D data calculation step S701, based on the correlation operation (Equation 3) between the exploration area obtained in the exploration area acquisition step S700 and the patch, the 2D data of the patch in the video frame at the current time t3 is calculated. However, I1 in Equation 3 is the exploration area, and I t is the patch, and each of them may be subjected to pixel normalization as in Equation 2 or enhancement processing on the patch. The process of obtaining the position of the patch within the exploration area based on the peak in the correlation distribution is the same as the process performed in the patch selection step S511. The value obtained by adding the center coordinate value of the exploration area to the patch position within the exploration area is used as the 2D data of the patch in the video frame at the current time t3. Also, similar to the patch selection step S511, Phase correlation or template matching can be used instead of the correlation operation.

[0050] The 2D data of the patch obtained in the 2D data calculation step S701 may be spurious values obtained due to the influence of misrecognition of the texture. In the peak determination S702, based on the height of the peak of the autocorrelation distribution of the patch, the correctness of the 2D data of the patch is determined. The reason why the peak height of the autocorrelation distribution can be used as a reference is the same as the reason described in the parallax reliability determination performed in the patch parallax calculation step S536 of the patch selection step S511.

[0051] If it is determined by the peak determination S702 that the 2D data of the patch is correct, the 2D data In the 2D data recording step S703, the 2D data is recorded in an array. For patches for which 2D data could not be obtained in the video frame at the current time t3, 2D data estimated values are calculated in the failed patch estimation step S517 described later. The 2D data recorded in the array and the 2D data estimated values are used in the estimation of the search area center coordinates in the search area acquisition step S700. In this way, the 2D coordinate data of the destination of the patch movement (that is, the 2D coordinates of each patch in the video frame at the current time) can be obtained by the patch tracking step S512.

[0052] <Regarding the 3D tracking step of the tracking step> An example of the procedure of the 3D tracking step S513 will be described with reference to FIG. 4C. In the 2D data interpolation step S750, when there is a slight deviation in the shooting time for each camera, the 2D data at the same shooting time is calculated by interpolation processing. An example (Figure 8) of the 2D data recorded in the array in the patch tracking step S512 will be used for explanation. In the above array, the 2D data of five cameras (referred to as cameras A, B, C, D, and E) at eight shooting times (t1, t1 + a, t1 + b, t1 + c, t2, t2 + a, t2 + b, t2 + c) is recorded. a and b are the shooting time offsets given to cameras B and C respectively, and c is the shooting time offset given commonly to cameras D and E. Each offset is shorter than the time interval from t1 to t2. The hyphens included in the 2D data indicate that the acquisition of the 2D data failed in the patch tracking step S512. Cameras A and B shoot the right eye, and cameras C, D, and E shoot near the left eye. Four patches are included for each of the right eye and left eye regions.

[0053] The calculation of the 2D data at time t2 will be described. Since the shooting time is different for each camera, at time t2, the 2D data of cameras other than camera A is not obtained. The 2D data of cameras other than camera A is calculated by interpolation processing (Equation 5) when the 2D data x bef , t aft (let's call it this) is obtained at times before and after time t2. bef , x aft Here, t

Equation

[0054] In the example of FIG. 8, 2D data at a time common to Cameras A, B, C, D, and E was calculated. However, for example, for Cameras A and B, 2D data at time t2, and for Cameras C, D, and E, 2D data at time t2 + b may be calculated. When the time is divided for each camera group, it is necessary to obtain the value of the same shooting time by interpolation processing in the following 3D data calculation step S751. Also, when the movement of the subject is slow and the movement within the shooting time offsets a, b, and c can be ignored, the interpolation process may not be executed, and 2D data at the time closest to t2 may be used instead.

[0055] In 3D data calculation step S751, based on the 2D data obtained by 2D data interpolation step S750, three-dimensional coordinate data of the patch (hereinafter referred to as 3D data) is calculated. For example, when two 2D data are obtained for one patch, 3D data P can be calculated using the following mathematical formula.

Equation

[0056] In Equation 6, a method that is robust against errors included in 2D data but has a high computational load (the least squares method using a pseudo-inverse matrix) is adopted. In a situation where no error is included, when backtracing two light rays from the three-dimensional coordinate data X1 and X2, a simplified process of obtaining the intersection of the two may be used. When three or more (denoted as N) pieces of 2D data are obtained for one patch, the procedure of selecting two out of the N pieces and obtaining 3D data by Equation 6 is repeated, and the average value of the multiple obtained 3D data can be calculated. The obtained 3D data is recorded in an array.

[0057] In the search presence / absence determination S752, it is determined whether the 3D data is being searched. In the case of the search state, the recorded 3D data is sent. In this way, the 3D tracking process S513 can calculate the 3D data of the patch.

[0058] <Regarding the subject movement calculation process in the tracking process> An example of the procedure of the subject movement calculation process S516 will be described with reference to FIG. 4D. In the common patch calculation process S770, the numbers (denoted as k = 1, 2,..., N) of the patches commonly included in the reference coordinate data and the 3D data (photographing time T) recorded in the array in the 3D tracking process S513 are acquired.

[0059] In the translation / rotation calculation process S771, using the reference coordinate data and the 3D data corresponding to the patch numbers obtained by the common patch calculation process S770, the translation / rotation of the subject at the photographing time T is calculated. The translation / rotation that minimizes the skin movement distance f given by the following equation is obtained.

Equation

[0060] When the above skin movement distance becomes the minimum value of 0, the translation and rotation accuracy of the subject is the highest. The translation and rotation obtained in the subject movement calculation step S516 are treated as approximate values with low accuracy. In this way, the subject movement calculation step S516 can calculate the approximate values of the translation and rotation, which are the subject movement data.

[0061] <Regarding the failed patch estimation step of the tracking process> An example of the procedure of the failed patch estimation step S517 will be described with reference to FIG. 4E. In the tracking failure determination S790, the 2D data recorded in the array in the patch tracking step S512 is examined to determine the presence or absence of a patch in which the 2D data is not recorded (hereinafter referred to as a failed patch). If a failed patch is confirmed, the 3D data estimation step S791 is executed.

[0062] In the 3D data estimation step S791, the 3D data of the failed patch is estimated using the following mathematical formula.

Equation

[0063] Here, although the 3D data X’ om of the failed patch is estimated based on the reference coordinate data X cm it is also possible to use a mathematical formula based on the 3D data X bef at a shooting time (denoted as T bm slightly before the current shooting time T).

Equation

[0064] In the 2D data estimation step S792, the 2D data within the reference plane image data for each camera is estimated by the following equation. bef

Equation

[0065] x’ is the 2D data estimated value of the failed patch for the i-th camera, and x’ is a vector with the first and second elements of the 3D data estimated value X’ cmi of the failed patch as components. u cm is a vector with the first and second elements of the aperture center coordinates U cm of the i-th camera as components. X’ i i (3) cm (3) i and U (3) i are the third elements of X’ cm and U i In this way, the two-dimensional coordinates of the patch that has moved outside the video frame or the patch for which the calculation has failed in the patch tracking step S512 can be estimated by the failed patch estimation step S517.

[0066] <Regarding the overall flow of the skin (subject) motion data creation process> An example of the procedure of the skin motion data creation process will be described with reference to FIG. 3B. In the grouping step S810, 3D data is acquired from the tracking process, and based on the skin movement distance f (Equation 7), the acquired 3D data is divided into a plurality of groups. In the in-group structuring step S811, 3D data representative of a group is selected, and translation and rotation parameters from the representative 3D data to group members are calculated.

[0067] In the optimization step S812, translation and rotation parameters between the reference coordinate data and the representative 3D data are calculated by an optimization process. Subsequently, based on the translation and rotation parameters obtained by the in-group structuring step S811 and the optimization step S812, the principal component vectors of the skin (subject) movement data may be calculated.

[0068] In the skin movement data transmission S813, the skin movement data, or the principal component vectors of the skin movement data, is transmitted to the tracking process. The processing of the skin movement data creation step corresponds to a process of evaluating the degree of similarity of the three-dimensional positional relationships of a plurality of small regions (patches) set on the surface of the subject by the skin movement distance f of Equation 7, and classifying local movements (skin (subject) movements) based on the degree of similarity.

[0069] <Regarding the grouping step of the skin movement data creation step> An example of the procedure of the grouping step S810 will be described with reference to FIG. 9A. In the 3D data presence / absence determination S830, for the 3D data acquired from the tracking process in the past, the presence or absence of 3D data not belonging to a group is determined. If there is no 3D data not belonging to a group, new 3D data is obtained by 3D data search S831 and 3D data reception S832. If there is 3D data not belonging to a group (hereinafter referred to as unassigned 3D data), the skin movement distance calculation step S833 is executed.

[0070] In the skin movement distance calculation step S833, the skin movement distance f (Equation 7) between the unassigned 3D data and one group member (3D data) for each group is calculated. X of Equation 7 ok 、Xck Put the unassigned 3D data and the 3D data of the group members into it respectively, and calculate the minimum value of the skin movement distance f. In this way, the skin movement distance of the unassigned 3D data for each group is calculated. Note that when no group has been created at the time of S833, the process of S833 may be skipped.

[0071] In the group update step S834, select the group with the minimum skin movement distance calculated in S833 as the candidate group. Then, compare the skin movement distance for the candidate group with a predetermined threshold. If the skin movement distance is smaller than the threshold, register the unassigned 3D data in the candidate group. If the skin movement distance for the candidate group is greater than or equal to the threshold, it is considered that there is no group close to the unassigned 3D data, generate a new group, and register the unassigned 3D data in the new group. Note that even when the process of S833 is skipped (when no group exists), a new group may be generated and the unassigned 3D data may be registered in the new group. In the 3D data number determination S835, determine the number of 3D data for which grouping has been completed. When the number of grouped 3D data reaches a certain number or more, end the grouping step S810.

[0072] The effect of the grouping step S810 will be described using the patch arrangement (FIGS. 10A and 10B). The 3D data obtained from the tracking process corresponds to the patches (pixel sections indicated by the markers 〇△□ in FIG. 10A) included in the video frames at the shooting times t = 0, 1,..., 8. The markers 〇△□ represent the differences in patch arrangement due to skin movement. After applying the grouping step S810, the video frames (3D data) at the shooting times t = 0, 1,..., 8 are grouped based on the differences in patch arrangement (markers 〇△□) due to skin movement (FIG. 10B). In this way, the grouping step S810 obtains a plurality of groups with similar magnitudes of skin movement included in the 3D data.

[0073] <Regarding the in-group structuring step of the skin movement data creation step> An example of the procedure of the in-group structuring process S811 will be described with reference to FIG. 9B. In the representative selection process S860, 3D data representing each group is selected. One of the group members may be selected, or translation and rotation (coordinate transformation) may be performed, and the average value of the group members with the minimum skin movement distance may be selected. Also, after selecting the group representative, the members may be replaced so that the skin movement distance from the new group representative becomes smaller, such as in a clustering process (e.g., the K-means method).

[0074] In the representative-member conversion calculation process S861, the conversion (translation and rotation parameters) from the group representative to each group member is calculated. For example, for X in Equation 7 ok , X ck , the 3D data of the group representative and the group member are respectively input, and the translation and rotation parameters that minimize the skin movement distance f are adopted.

[0075] In the decimation process S862, groups with extremely fewer group members compared to other groups are deleted. Such groups are likely to be caused by errors in the 3D data and may cause errors in the subject movement measurement values.

[0076] Note that when the group member used in the calculation in the skin movement distance calculation process S833 is used as the group representative, the representative selection process S860 can be omitted. Similarly, in the skin movement distance calculation process S833, the representative-member conversion calculation process S861 can also be omitted by using the translation and rotation parameters obtained when calculating the skin movement distance (Equation 7).

[0077] The effect of the in-group structuring process S811 will be described with reference to FIG. 11. In the representative selection process S860, for three groups, the 3D data at the shooting times t = 0, 2, and 6 are selected as representatives. Since the members of each group are patch arrangements with translation and rotation given to the group representative, the translation and rotation parameters are obtained by the representative-member conversion calculation process S861. In this way, the coordinate transformation parameters for group members are obtained from the 3D data representing each group by the in-group structuring process S811.

[0078] <Regarding the optimization process of the skin motion data creation process> An example of the procedure of the optimization process S812 will be described with reference to FIG. 9C. In the inter-group structuring process S880, an approximation value of the transformation (translation and rotation parameters) from the reference coordinate data selected based on the 3D data to the 3D data representing each group is calculated. The reference coordinate data may be any 3D data, but if one of the data representing each group is selected, the calculation of the approximation value of the translation and rotation parameters can be omitted for one group. The reason for expressing it as an approximation value is that the skin motion distance does not approach 0 between the 3D data included in different groups, and accurate translation and rotation parameters cannot be calculated.

[0079] In the parameter adjustment process S881, the approximation values of the translation and rotation parameters obtained in the inter-group structuring process S880 are adjusted so that the evaluation function φ given by the following mathematical formula is minimized.

Equation

Equation

Equation

[0080] The evaluation function φ in Equation 11 is a function created based on the properties of coordinate transformation with respect to the coordinate axes of the reference coordinate data (hereinafter referred to as the reference coordinate axes). An example (FIGS. 12 and 13) of projecting the reference coordinate axes into the video frame using the coordinate transformation between 3D data will be used for explanation. Here, the reference coordinate data is used as the representative value of group 0. The symbol 887 indicates the reference coordinate axes. The translation / rotation parameter 885 is a value obtained by the in-group structuring process S811, and the translation / rotation parameter approximation 886 is a value obtained in the between-group structuring process S880. By the coordinate transformation using the translation / rotation parameter 885 and the translation / rotation parameter approximation 886, the reference coordinate axes 888 after coordinate transformation are obtained in each video frame.

[0081] When the reference coordinate axes 888 after the coordinate transformation shown in FIG. 12 are rearranged in time series, FIG. 13 is obtained. Therefore, if the translation / rotation parameter approximation 886 is adjusted to the correct value, the temporal changes in the origin and direction of the reference coordinate axes 888 after coordinate transformation will be the smoothest.

[0082] The function diff_s given by Equation 12 represents the difference in the origin coordinates of the reference coordinate axes 888 after coordinate transformation between adjacent imaging times. On the other hand, the function diff_q given by Equation 13 represents the difference in the three-dimensional rotation of the reference coordinate axes 888 after coordinate transformation. Since the evaluation function φ of Equation 11 is the sum of the functions diff_s and diff_q, it represents the magnitude of the change in the reference coordinate axes 888 after coordinate transformation. Thus, if the minimization of the evaluation function φ is targeted, the time-series change in the reference coordinate axes 888 after coordinate transformation becomes small, and the translational-rotation parameter approximation 886 is adjusted to the correct value. Note that since there are also indicators representing the difference in the coordinate system origin and three-dimensional rotation other than Equations 12 and 13, other equations may be used.

[0083] Various optimization methods can be used to adjust the translational-rotation parameter approximation. However, since the evaluation function φ of Equation 11 is in the form of a sum of squares and has a large number of variables, it is suitable to use a multivariable nonlinear least squares method such as the Levenberg-Marquardt method.

[0084] The structured process S880 between groups can be executed using patches or feature points to obtain an accurate translational-rotation parameter approximation. In that case, the parameter adjustment process S881 may be omitted.

[0085] In the skin motion data calculation process S882, the inverse transformation of the coordinate transformation from the reference coordinate data to each 3D data is obtained by the following equation, and the 3D data on the reference coordinate axes 887 (i.e., skin motion data) is calculated.

Equation

[0086] Skin movement data X (t) zk The shooting time t of [] can be arbitrarily selected, but it is necessary to sample so as to cover all states of skin movement. For example, if the shooting times corresponding to the representative 3D data of all groups are selected, all states of skin movement can be covered.

[0087] [First Embodiment] The subject movement measurement device 1000 according to the first embodiment of the present invention will be described with reference to FIG. 14A. FIG. 14A is a diagram showing an example of the configuration of the subject movement measurement device 1000 according to the first embodiment. The subject movement measurement device 1000 according to the present embodiment includes four cameras 1001, a camera height adjustment stage 1002, a white light source 1003, a reflector 1004, and a chair 1005. Further, the subject movement measurement device 1000 includes a computer 1006, a display 1007, a mouse 1008, a keyboard 1009, a network cable 1010, and a display cable 1011. The computer 1006 is connected to the LAN 1016 via the switching hub 1015.

[0088] The four cameras 1001 are connected to the computer 1006 by the network cable 1010. The camera 1001 shoots moving image data of 1 megapixel at a frame rate of 60 fps, and the moving image data is transferred to the computer 1006. The camera 1001 is adjusted so that the vicinity of the eyes of the subject 1014 is in focus with the subject 1014 with the back of the head against the backrest of the chair 1005. The height of the camera 1001 can be adjusted by the camera height adjustment stage 1002. Also, the camera 1001 is rotated 90 degrees so that the long side of the video frame is vertical. This is to widen the distance in the vertical direction (from the nose to the mouth) between the patches and improve the measurement accuracy of the rotation due to the nodding motion. The white light source 1003 uniformly illuminates the reflector plate 1004, and the reflected light 1012 from the reflector plate 1004 is irradiated onto the head of the subject 1014.

[0089] The display 1007 is connected to the computer 1006 by a display cable 1011. FIG. 14B is an example of a display screen 1020 displayed on the display 1007, and the state of the camera 1001, calculation content, etc. are displayed. On the display screen 1020, for example, a video display section 1021, an information display / modification section 1022, a list box 1023, a check box 1024, an edit box 1025, and an instruction button 1026 can be displayed.

[0090] <Flow of processing in the subject movement measurement device 1000> Subsequently, the flow of processing performed by the subject movement measurement device 1000 will be described with reference to FIGS. 15A and 15B. The operator 1013 of the subject movement measurement device 1000 first selects a setting 1031, which is an item in the list box 1023, in order to specify information (hereinafter referred to as the display mode) to be superimposed and displayed on the video display section 1021 (FIG. 15A). As a result, a target mark is superimposed and displayed at the position where the eyes should be in the video display section 1021. The operator 1013 instructs the subject 1014 to sit with their head against the backrest of the chair 1005, and adjusts the camera height adjustment stage so that the eyes of the subject 1014 appear near the target mark in the video display section 1021. After the adjustment is completed, the operator 1013 inputs the instruction button 1026 regarding the completion of adjustment 1070 (FIG. 15B).

[0091] ​Next, the operator 1013 uses the check box 1024 to input the measurement data type 1040 and the display data type 1041 (Fig. 15B). Also, the operator 1013 uses the edit box 1025 to input the measurement frequency 1042, the display frequency 1043, the measurement time 1044, the delay time 1045, the storage destination 1046, and the connection destination 1047 (Fig. 15B). The measurement data type 1040 and the display data type 1041 include the subject movement data 1060 (skin movement data 1061), the patch position 1062, the number of blinks 1063, and the line-of-sight direction 1064. After all items are input, the instruction button 1026 regarding measurement start 1071 becomes clickable.

[0092] When the instruction button 1026 regarding measurement start 1071 is input, the tracking process is started in the computer 1006. The subsequent steps are as described in the explanation of the subject movement measurement method.

[0093] During the execution of the tracking process, data 1032 is selected in the list box 1023 regarding the display mode (Fig. 15A). As a result, the data of the type input as the display data type 1041 is displayed on the video display unit 1021.

[0094] The data specified by the measurement data type 1040 is sent from the tracking process to the storage destination 1046 and the connection destination 1047 as soon as the calculation result is obtained. Here, when the delay time 1045 is short, the difference between the shooting time and the time when the calculation result is obtained may exceed the delay time 1045. In this case, based on the calculation result of the previous shooting time, the data at the shooting time is estimated, and the estimated value is sent to the storage destination 1046 and the connection destination 1047.

[0095] When the instruction button 1026 regarding measurement end 1072 is input, an end instruction is sent to the tracking process, and the measurement ends.

[0096] Thus, the first embodiment of the present invention is a subject movement measurement device that can measure subject movement with high accuracy.

[0097] [Second Embodiment] The subject motion measurement device 1100 according to the second embodiment of the present invention will be described with reference to FIG. 16. FIG. 16 is a diagram showing an example of the configuration of the subject motion measurement device 1100 according to the second embodiment. The subject motion measurement device 1100 according to the present embodiment realizes an improvement in accuracy, a higher frame rate, and a smaller size of the subject motion measurement device 1000 of the first embodiment. Since the basic configuration is the same as that of the subject motion measurement device 1000, mainly the differences will be described.

[0098] The functions corresponding to the camera 1001, the illumination 1003, and the computer 1006 of the first embodiment are integrated in the subject motion measurement device 1100. Functions such as display and information input are realized by the tablet PC 1101. Communication between the subject motion measurement device 1100 and the tablet PC 1101 is performed by wireless communication 1103 and is connected to the LAN 1016 via the wireless router 1102. The subject motion measurement device 1100 is installed in front of the face of the subject 1014 and used (for example, it may be attached to a support mechanism such as the camera height adjustment stage 1002 in FIG. 14A).

[0099] The subject motion measurement device 1100 according to the present embodiment is composed of four cameras 1105, a near-infrared illumination 1106, a near-infrared spot array illumination 1107, and a control unit 1108. The cameras 1105, the near-infrared illumination 1106, and the near-infrared spot array illumination 1107 are connected to the control unit 1108 by a cable 1109.

[0100] <Configuration of Camera and Illumination> The configuration of the camera and illumination in the subject motion measurement device 1100 will be described with reference to FIGS. 17A to 17D. The four cameras 1105 are arranged like the cameras 1120, 1121, 1122, and 1123 in FIG. 17A. The cameras 1120 and 1122 capture moving images near the left eye 1125, and the cameras 1121 and 1123 capture moving images near the right eye 1126.

[0101] The near-infrared illumination 1106 is arranged like the near-infrared illuminations 1130 and 1131 in FIG. 17B. The near-infrared spot array illumination 1107 is arranged like the near-infrared spot array illuminations 1132 and 1133. The near-infrared illumination 1130 and the near-infrared spot array illumination 1132 illuminate the vicinity of the left eye 1125, and the near-infrared illumination 1131 and the near-infrared spot array illumination 1133 illuminate the vicinity of the right eye 1126.

[0102] As shown in FIG. 17C, an angular difference is given to the optical axis so that the imaging range of the camera overlaps with the illumination ranges of the near-infrared illumination and the near-infrared spot array illumination. The near-infrared illumination uniformly illuminates the vicinity of each eye, while the near-infrared spot array illumination creates spots arranged on a two-dimensional grid point as shown in FIG. 17D.

[0103] The illumination light wavelength in the near-infrared illumination 1106 can be arbitrarily selected from the near-infrared wavelength band. However, if the absorption wavelength of the pigment (such as melanin) contained in the subject is selected, the contrast of the texture within the patch is improved, and the tracking accuracy in the patch tracking steps S512, S900, and S910 is improved.

[0104] On the other hand, the illumination light wavelength of the near-infrared spot array illumination 1107 can also be arbitrarily selected from the near-infrared wavelength band, but a wavelength with less absorption is suitable so that the brightness of the spots is uniform.

[0105] As will be described later, the near-infrared spot array illumination 1107 is used in combination with the near-infrared illumination 1106 when executing the patch selection step S511. Only the near-infrared illumination 1106 is used in other steps.

[0106] <Configuration of the control unit> The configuration of the control unit 1108 in the subject movement measurement device 1100 will be described with reference to FIG. 18. The control unit 1108 includes a wireless communication interface 1150, a camera interface 1151, an illumination interface 1152, a controller (processing unit) 1153, a data bus 1154, a memory 1155, and a plurality of processing elements 1157.

[0107] The processing element 1157 is a circuit that executes part of the processing of the subject movement measurement method 500. As the plurality of processing elements 1157, a calibration unit 1160, a patch selection unit 1161, a patch tracking unit 1162, a 3D tracking unit 1163, a head movement calculation unit 1164, and a failed patch estimation unit 1165 are provided. Further, as the plurality of processing elements 1157, a grouping unit 1170, an in-group structuring unit 1171, and an optimization unit 1172 are provided. The processing executed by these processing elements 1157 corresponds to the steps of the same name. For example, the optimization unit 1172 executes the optimization step S812. In the example of the subject movement measurement method 500, common processing occupies most of the subject movement calculation steps S516 and S902, but there may be slightly different processing. The processing element 1157 can execute such a plurality of slightly different processes.

[0108] The controller (processing unit) 1153 is connected to the wireless communication interface 1150, the camera interface 1151, the lighting interface 1152, the data bus 1154, the memory 1155, and the processing element 1157 via the signal line 1156. Also, the wireless communication interface 1150, the camera interface 1151, the lighting interface 1152, the memory 1155, and the processing element 1157 are also connected via the data bus 1154. Communication with the controller (processing unit) 1153 is performed using the signal line 1156, and a large amount of data is exchanged using the data bus 1154.

[0109] The controller (processing unit) 1153 can execute the procedure of the subject movement measurement method 500 by repeating the calculation execution instruction to the processing element 1157 and the connection destination instruction to the data bus 1154. That is, the subject movement measurement device 1100 has a controller (processing unit) 1153 that acquires three-dimensional coordinate data of patches set in the first image data and the second image data based on the parallax between the first image data and the second image data captured by a plurality of cameras. Then, the controller (processing unit) 1153 can calculate the movement data of the subject from the three-dimensional coordinate data of the patch and the reference coordinate data serving as a reference for the subject. The controller (processing unit) 1153 calculates the translation and rotation of the subject from the three-dimensional coordinate data of the patch and the reference coordinate data serving as a reference for the subject.

[0110] <Flow of processing in the subject movement measurement device 1100> The processing of the subject movement measurement device 1100 is mostly the same as that of the subject movement measurement device 1000 in the first embodiment, but the processing performed by the patch selection unit 1161 is different from the patch selection step S511 shown in FIG. 4A.

[0111] In the patch selection unit 1161, instead of the patch selection step S511 in FIG. 4A, a method using the near-infrared spot array illumination 1107 is executed. An example of the processing executed by the patch selection unit 1161 will be described with reference to FIG. 19.

[0112] In the acquisition of video frames S1190, while the near-infrared illumination 1106 is lit, shooting is performed while switching the lighting of the near-infrared spot array illumination 1107, and a plurality of frames are acquired for each camera.

[0113] The black-eye detection step S1191, the contrast detection step S1192, and the selectable region extraction step S1193 are the same as the black-eye detection step S531, the contrast detection step S532, and the selectable region extraction step S533 in FIG. 4A. However, it is necessary to use only the video frames in which the near-infrared spot array illumination 1107 is not lit. As a result of the selectable region extraction step S1193, reference plane image data is obtained.

[0114] In the corresponding point calculation step S1194, first, a video frame that is close to the reference plane image data in terms of shooting time and includes the pattern of the near-infrared spot array illumination 1107 is selected. Subsequently, by counting the number of spots from the edge of the selected video frame, each spot is numbered.

[0115] In the patch calculation step S1195, patch candidates near spots with the same number are extracted from a plurality of reference plane image data obtained from different cameras. Among the obtained patch candidates, candidates with high contrast are adopted as patches.

[0116] When using the near-infrared spot array illumination 1107, there is no problem even in a state where the deformation applied to the patch due to the influence of the surface normal is different for each camera (Fig. 7B). Since the options for patch candidates expand it is possible to adopt a patch with high contrast and improve the tracking accuracy.

[0117] Here, the three-dimensional shape measurement method in which spot array illumination is added to a camera is known as the active stereo method. For example, it is possible to acquire the three-dimensional shape (depth map) near the left eye 1125 using cameras 1120, 1122 and the near-infrared spot array illumination 1132. The surface normal of the patch candidate may be calculated from the depth map, and a patch whose surface normal is close to the reference surface normal (the change in the patch shape when the subject moves is small) may be adopted. Also, in this embodiment, a spot array pattern is used for illumination, but any pattern may be used as long as the position information within the pattern can be acquired by different cameras.

[0118] In this way, the second embodiment of the present invention utilizes the wavelength and pattern of illumination light, and further realizes dedicated circuitization of the computing function to perform high-load calculations, thereby simultaneously achieving improved accuracy, higher frame rate, and miniaturization, and providing a subject motion measurement device.

[0119] [Third Embodiment] The imaging system 1300 according to the third embodiment of the present invention will be described with reference to FIG. 20. FIG. 20 is a diagram showing an example of the configuration of the imaging system 1300 according to the third embodiment. The imaging system 1300 according to the present embodiment includes a magnetic resonance imaging apparatus 1301 and a subject movement measurement apparatus 1302. The magnetic resonance imaging apparatus 1301 and the subject movement measurement apparatus 1302 are connected by a network cable 1303 and can communicate with each other. Further, the magnetic resonance imaging apparatus 1301 and the subject movement measurement apparatus 1302 are connected to the LAN 1305 via the network cable 1303 and a switching hub 1304.

[0120] Note that the imaging system 1300 according to the present embodiment can be applied to any modality capable of imaging a subject. The imaging system 1300 can be applied to a single modality such as a magnetic resonance imaging (MRI) apparatus or an X-ray computed tomography (CT) apparatus. Further, the imaging system 1300 can also be applied to a single modality such as a positron emission tomography (PET) apparatus. Furthermore, the imaging system 1300 can also be applied to a single modality such as a single photon emission computed tomography (SPECT) apparatus. Alternatively, the imaging system according to the present embodiment may be applied to a combined modality such as an MR / PET apparatus, a CT / PET apparatus, an MR / SPECT apparatus, or a CT / SPECT apparatus. However, for the sake of specific description below, it is assumed that the imaging system according to the present embodiment is a magnetic resonance imaging apparatus (MRI apparatus).

[0121] <Configuration of the magnetic resonance imaging apparatus 1301> The magnetic resonance imaging apparatus 1301 will be described with reference to Fig. 21. The magnetic resonance imaging apparatus 1301 is composed of a static magnetic field magnet 1311, a gradient magnetic field coil 1312, an RF coil 1313, a head coil 1314, a bed 1315, a tabletop 1316, an MRI controller 1317, a cable 1318, and a white light 1319. The MRI controller 1317 is connected to the gradient magnetic field coil 1312, the RF coil 1313, the head coil 1314, and the bed 1315 via the cable 1318.

[0122] The bed 1315 has a top plate 1316 that can be moved up and down and back and forth. An MRI controller 1317 controls the bed 1315 and carries in and out a subject 1321.

[0123] In addition, the MRI controller 1317 controls the gradient magnetic field by the gradient magnetic field coil 1312. The MRI controller 1317 has a built-in board computer and generates magnetic resonance data and magnetic resonance image data based on the magnetic resonance signals. The MRI controller 1317 is connected to the LAN 1305 using the network interface of the board computer. It is capable of receiving imaging instructions from software on a PC connected to the LAN and transmitting magnetic resonance data and magnetic resonance image data to the PC.

[0124] The subject motion measurement device 1302 includes an illumination device 1319 that illuminates the subject 1321. The illumination device (white illumination) 1319 illuminates the bore inner wall 1320 of the gantry that photographs the subject 1321, and illuminates the subject 1321 with the reflected light 1338 on the bore inner wall 1320. The illumination device 1319 is installed on the bore inner wall 1320. The illumination device 1319 may be installed on the top plate 1316, a frame (support) 1332 described later, etc. The illumination device 1319 is installed so as to illuminate the ceiling (upper side of the drawing) on the bore inner wall 1320 obliquely. The illumination device 1319 is installed so as to illuminate the area of the bore inner wall 1320 in front of the head of the subject 1321. The illumination device 1319 is installed so as to illuminate the subject 1321 with indirect illumination on the bore inner wall 1320 from the gap of the head coil 1314. The brightness of the reflected light 1338 is about the level of indoor light. Since the illumination device 1319 is indirect illumination that illuminates the bore inner wall 1320, the subject 1321 can be illuminated uniformly.

[0125] Also, by adjusting the illumination light of the illumination device 1319, the first image data and the second image data captured by a plurality of cameras can be captured as images with the same brightness. That is, the first image data and the second image data captured by a plurality of cameras will be captured as images with the same brightness. There are irregularities on the surface of the skin of the subject 1321. If the subject 1321 is illuminated with direct illumination, unevenness in brightness will occur, and areas that become shadows are likely to appear. Here, the illumination device 1319 is arranged to be indirect illumination so that the areas that become shadows are reduced.

[0126] Also, if the subject 1321 is illuminated with direct illumination, the eyes of the subject 1321 will be directly illuminated, and the subject 1321 will feel dazzled. However, since the illumination device 1319 is indirect illumination that illuminates the bore inner wall 1320, the subject 1321 can be photographed without feeling dazzled.

[0127] The controller (processing unit) 1153 can also perform image processing on the first image data and the second image data so that they have the same brightness. Specifically, the controller (processing unit) 1153 performs histogram analysis based on brightness on the first image data and the second image data. The controller (processing unit) 1153 obtains the maximum brightness values in the histogram of the first image data and the histogram of the second image data, respectively. The controller (processing unit) 1153 adjusts the brightness of the first image data or the second image data so that the maximum brightness values of the respective brightnesses match, based on the maximum brightness value of each. In this way, by making the brightnesses of the first image data and the second image data the same, respective image recognition can be performed equally, and tracking can be appropriately performed.

[0128] <Configuration of the subject movement measurement device 1302> The subject movement measurement device 1302 will be described with reference to FIG. 21. The subject movement measurement device 1302 includes four cameras 1330, a near-infrared illumination 1331, a frame (support) 1332, a pole 1333, a mirror 1334, a fixture 1335, and a subject measurement device controller 1336. The subject measurement device controller 1336 is connected to each camera 1330 by a network cable 1303. Communication is possible between the subject measurement device controller 1336 and each camera 1330.

[0129] The subject measurement device controller 1336 incorporates a board computer and can execute the software of the subject movement measurement method 500. The board computer of the subject measurement device controller 1336 is connected to the MRI controller 1317 and can send various measurement data at specified timings. It can also receive instructions from the MRI controller 1317. Note that the subject measurement device controller 1336 does not necessarily have to incorporate a board computer. Any device can be used as long as it can execute the software of the subject movement measurement method 500. For example, the subject measurement device controller 1336 has, as hardware resources, processors such as a CPU, GPU, and MPU, and memories such as a ROM and a RAM. Note that it may be realized by an Application Specific Integrated Circuit (ASIC). Alternatively, it may be realized by a Field Programmable Logic Device (FPGA). Alternatively, it may be realized by another Complex Programmable Logic Device (CPLD). Alternatively, it may be realized by a Simple Programmable Logic Device (SPLD).

[0130] The frame 1332 is equipped with tires 1337 and can move on the top plate 1316 of the bed 1315. Also, the frame 1332 can be fixed at a fixed position on the top plate 1316 by a fastener 1335.

[0131] The near-infrared illumination 1331 compensates for the brightness of the white illumination 1319 and enhances the contrast of the texture included in the patch.

[0132] The relationship between the arrangement of the four cameras 1330 and the opening of the head coil 1314 to be imaged will be described with reference to FIGS. 22A and 22B. The high-resolution head coil 1314 has a large number of built-in coils, and the closer the coils are to the head, the higher the signal-to-noise ratio. Since it is necessary to lay out the coils in a limited area near the head, the head coil 1314 needs to cover the entire head of the subject. In addition, since the outer wall of the head coil 1314 does not transmit light, it is difficult to take images from outside the opening. However, when the eyes and mouth are covered, the pressure felt by the subject is very strong, so the right-eye opening 1350, the left-eye opening 1351, and the mouth opening 1352 are usually provided (FIG. 22A).

[0133] Since the jaw and the mouth area move independently of the skull, the imaging results of the mouth opening 1352 are not suitable for measuring the movement of the subject (movement of the skull). Therefore, the four cameras 1330 are used to image the area near the eyes of the subject through the right-eye opening 1350 and the left-eye opening 1351 (FIG. 22B).

[0134] When measuring the movement of the subject from a narrow range, the measurement accuracy of the rotation angle due to the head shaking motion is likely to decrease. In order to improve the measurement accuracy, it is necessary to increase the distance between the patches within the video frame. If the imaging magnification is increased, the distance between the patches will increase, but the moving speed of the patches within the video frame will also increase, resulting in motion blur. In order to take the distance between the patches while avoiding motion blur, the imaging magnification of the cameras 1330 is set so that the entire image of each of the right-eye opening 1350 and the left-eye opening 1351 is captured within the video frame. In addition, the distance between the horizontal patches can be increased by using the distance between the right eye and the left eye. In order to increase the distance between the vertical patches, it is necessary to set the long side of the video frame in the vertical direction. Therefore, each of the cameras 1330 is rotated 90 degrees around the optical axis and fixed to the frame 1332.

[0135] <Flow of processing in the imaging system 1300> During the operation of the imaging system 1300, the magnetic resonance imaging device 1301 is at any time Various measurement data can be acquired from the subject movement measurement device 1302. For example, after acquiring subject movement data from the subject movement measurement device 1302, highly accurate magnetic resonance image data can be generated by correcting the subject movement through the control of the gradient magnetic field coil 1312.

[0136] In the imaging system 1300, the magnetic resonance imaging device 1301 and the subject movement measurement device 1302 are connected by the network cable 1303, but another connection method may also be used. For example, in the board computer of the MRI controller 1317, the software program of the subject movement measurement method 500 may be executed. In this case, it is necessary to connect the board computer of the MRI controller 1317 to the camera 1330. Furthermore, when the MRI controller 1317 is equipped with an FPGA board instead of a board computer, an image processing circuit such as the control unit 1108 may be implemented on the FPGA board.

[0137] As described above, the third embodiment of the present invention can provide an imaging system that can provide highly accurate subject movement information and related information to the magnetic resonance imaging device, and can improve the image quality and added value of the magnetic resonance image data.

Explanation of Reference Numerals

[0138] 1000, 1100, 1302: Subject movement measurement device

Claims

1. A processing unit that acquires three-dimensional coordinate data of a patch set for image data including a predetermined part of the subject and a peripheral part of the predetermined part, which are acquired from an opening of the head coil in frame data included in moving image data obtained by photographing the subject with a camera while the subject is wearing the head coil; An estimation unit that estimates the position of the patch based on reference coordinate data serving as a reference for the subject, and has, The processing unit tracks the patch in the moving image data captured by the camera, and calculates movement data of the head of the subject based on the three-dimensional coordinate data of the tracked patch. The estimation unit estimates the position of the patch when the patch moves outside the frame of the frame data, and a subject movement measurement device characterized by this.

2. The moving image data is captured using a plurality of cameras, The processing unit acquires three-dimensional coordinate data of the patch based on a parallax between first image data and second image data captured by the plurality of cameras, and the subject movement measurement device according to claim 1, characterized by this.

3. The processing unit calculates translation and rotation of the subject from the three-dimensional coordinate data of the patch and reference coordinate data serving as a reference for the subject, and the subject movement measurement device according to claim 1, characterized by this.

4. The moving image data is captured using a plurality of cameras, The first image data and the second image data captured by the plurality of cameras are image data captured from different positions, and the subject movement measurement device according to claim 1, characterized by this.

5. The patches included in the first image data and the second image data are the same size and the same shape, respectively, and the subject movement measurement device according to claim 2 or 4, characterized by this.

6. The first image data and the second image data are image data captured from different directions of a peripheral part of a predetermined part of the subject, and the subject movement measurement device according to claim 2 or 4, characterized by this.

7. The predetermined part is a part that can be photographed by the plurality of cameras from a gap of a receiving coil attached to the subject and receiving a magnetic resonance signal, and the subject movement measurement device according to claim 6, characterized by this.

8. The predetermined part is the eye or nose of the subject, and the subject movement measurement device according to claim 6, characterized by this.

9. The subject movement measurement device according to claim 1, wherein a plurality of the patches are provided in the frame data, and the plurality of patches are provided at a distance of a predetermined distance or more from each other.

10. The subject movement measurement device according to claim 1, wherein the patch is set in a band region in the frame data.

11. The subject movement measurement device according to claim 10, wherein the processing unit normalizes pixel values in the band region.

12. The subject movement measurement device according to claim 1, further comprising an illumination device that illuminates the bore inner wall of the gantry and illuminates the subject with reflected light on the bore inner wall.

13. The moving image data is captured using a plurality of cameras, The subject movement measurement device according to claim 12, wherein the first image data and the second image data captured by the plurality of cameras are captured as images having the same brightness by adjusting illumination light from the illumination device.

14. The subject movement measurement device according to claim 9, further comprising an estimation unit that estimates the position of a patch that has moved outside the frame based on patches within the frame when the patch has moved outside the frame of the frame data.

15. Obtaining three-dimensional coordinate data of a patch set for image data including a predetermined part of the subject and a peripheral part of the predetermined part, obtained from an opening of the head coil in frame data included in moving image data captured by a camera with the subject wearing a head coil; Tracking the patch in the moving image data captured by the camera, and calculating movement data of the head of the subject based on the three-dimensional coordinate data of the tracked patch; When the patch has moved outside the frame of the frame data, estimating the position of the patch based on reference coordinate data serving as a reference for the subject; A subject movement measurement method, characterized by comprising:

16. A program for causing a computer to execute each step of the subject movement measurement method according to claim 15.

17. An imaging system for imaging a subject, The subject movement measurement device according to any one of claims 1 to 13 for measuring the movement of the subject, an imaging device that corrects the motion of the subject measured by the subject motion measurement device and images the subject; An imaging system comprising:

18. 20. The imaging system of claim 17, wherein the imaging device is a magnetic resonance imaging device.

19. 2. The subject movement measuring device according to claim 1, wherein the patch is set on the periphery of the predetermined portion.

20. 2. The subject movement measuring device according to claim 1, wherein the specified portion is a portion that is linked to local movement on the surface of the subject, and the peripheral portion of the specified portion is a portion that is linked to movement of the head.

21. 2. The subject movement measuring apparatus according to claim 1, wherein the head movement data is used to correct a gradient magnetic field.

Images