Measurement processing terminal, method, and computer program for measuring and processing finger movements

Abstract

Provided are a measurement processing terminal, method, and computer program with which it is possible to not only capture finger joint movements and quantitatively evaluate the finger flexion / extension function and / or opening and closing movements of two fingers, but also evaluate the upper extremity motor function in an objective, detailed, accurate, and precise manner.　This measurement processing terminal 50 has: an imaging data collector 9 for collecting imaging data obtained by imaging the movements of the fingers of a subject; a hand-tracking data generator 26 equipped with a hand-tracking function for detecting and tracking the positions of the fingers on the basis of the imaging data, the hand-tracking data generator 26 generating time-based hand-tracking data by means of the hand-tracking function from the imaging data; and a data processor 27 for processing the hand-tracking data obtained from the hand-tracking data generator so as to generate quantitative data relating to finger flexion / extension and / or opening and closing movements of two fingers accompanying finger joint movements, as well as for processing hand-tracking data, time data and distance data obtained from a movement distance measurement unit, and line-of-sight data obtained from a line-of-sight detector so as to generate correlation data in which correlation between the above items of data is quantified.

Description

[Technical field]

[0001] The present invention relates to a measurement processing terminal, a measurement processing method, and a computer program for measuring finger movements, including finger tapping movements, and processing the measurement results. [Background technology]

[0002] As the aging of society progresses, the number of Alzheimer's disease patients is increasing year by year, and if it is detected early, the progression of the disease can be slowed down with medication. However, it can be difficult to distinguish between symptoms of aging, such as forgetfulness, and the disease itself, and in many cases, patients only visit a hospital once their condition has progressed to a severe stage.

[0003] In this situation, screening tests for the early detection of Alzheimer's disease have traditionally included blood tests, olfactory tests, and tests that replicate a doctor's interview on a tablet device, but these have problems with the pain of blood sampling and the length of time required for the test. On the other hand, cognitive function assessments using button presses or one-handed finger movement measurements using a tablet device have also been performed as tests that place less of a burden on the subject, but they have the drawback of not being able to achieve sufficient test accuracy. If a simple screening test could be performed that is highly accurate and places less of a burden on the subject, it could lead to the early detection of Alzheimer's disease, improve the quality of life of patients, and contribute to reducing medical and nursing care costs.

[0004] In recent years, it has become clear that a movement pattern specific to Alzheimer's dementia can be extracted from the opening and closing movement of two fingers (finger tapping movement) using the thumb and index finger of both hands, and it has been confirmed that there is a high correlation with finger movement measurement and dementia tests by general interview. It is said that these are the results of capturing the decline in rhythmic motor function of the fingers of both hands caused by brain atrophy in Alzheimer's dementia by finger tapping movement measurement. In addition, the fingers are said to be the second brain, and many areas of the brain are related to the function of the fingers, and finger movements are said to be related not only to Alzheimer's dementia, but also to dementia such as cerebrovascular and Lewy body dementia, Parkinson's disease, and developmental coordination disorder (such as being unable to skip or jump rope). In other words, it is possible to know the state of the brain from finger tapping movement. Furthermore, finger tapping movement can be used as a "ruler" that indicates the health state of the brain, making it possible to quantify the fine motor function of the fingers, so it can be used in various fields such as healthcare, rehabilitation, and life support.

[0005] As a method for measuring and evaluating finger tapping motion with high accuracy, for example, Patent Document 1 and Patent Document 2 disclose a motor function evaluation system and method including a motor function measuring device that calculates motor data based on the relative distance between a pair of a transmitter coil and a receiver coil attached to a moving part of a living body, and an evaluation device that evaluates the motor function of a living body based on the motor data received from the motor function measuring device. That is, these patent documents disclose that a magnetic sensor attached to the fingertips converts the change in magnetic force that fluctuates due to the tapping motion of two fingers into an electric signal, and the movement is measured and quantified to capture a feature value that indicates the characteristics of the finger movement, thereby knowing the state of brain function.

[0006] Additionally, in the field of rehabilitation, the Simple Test of Upper Limb Function (STEF) is used to check the effectiveness of postoperative rehabilitation for stroke and cerebral infarction patients, in which the time required for a series of movements to grasp and move objects of different sizes and shapes is measured with a stopwatch. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2016-49123 A [Patent Document 2] JP 2015-217282 A Summary of the Invention [Problem to be solved by the invention]

[0008] However, the finger tapping device using the magnetic sensor attached to the fingertip as disclosed in the above-mentioned patent document cannot recognize the movement of the finger joints, and therefore cannot quantitatively evaluate the finger flexion / extension movement such as "pinching" motion. In addition, if it is difficult to attach the sensor due to injury or deformation of the fingers, it is not possible to perform measurement.

[0009] In addition, the Simplified Test of Upper Extremity Function (STEF) mentioned above involves a doctor visually observing the movements of the arms and fingers, so there is a large degree of subjectivity in the hands of the person measuring (the doctor), and the test results may differ depending on the person measuring.

[0010] Furthermore, in order to objectively, precisely, and accurately evaluate upper limb motor function, it is necessary to evaluate not only the fingers but also the movements of other parts of the body, such as the arms and eyes, in synchronization with each other (for example, quantitatively evaluating the coordination (relationship) between the movements of the fingers and arms, and the coordination (relationship) between the movements of the fingers and eyes). However, with conventional testing methods that involve visual confirmation, it is difficult to simultaneously evaluate the movements of different parts of the body, as this places a heavy burden on the person measuring.

[0011] Furthermore, in order to accurately grasp the degree of recovery of upper limb motor function or to prevent variations in testing and measurement, it is necessary to standardize the testing and measurement environment and make the testing and measurement conditions uniform. Depending on the testing and measurement method, if such testing and measurement conditions (or environment) are not standardized, there may be cases where the testing and measurement results vary between subjects and accurate testing and measurement cannot be performed.

[0012] The present invention has been made in consideration of the above circumstances, and aims to provide a measurement processing terminal, method, and computer program that can capture the movement of finger joints and quantitatively evaluate the flexion / extension motor function of fingers and / or the opening and closing motor function of two fingers, as well as objectively, precisely, and with high accuracy the motor function of the upper limbs. [Means for solving the problem]

[0013] In order to solve the above-mentioned problems, the present invention provides a measurement processing terminal that measures the movement of a subject's fingers and processes the measurement results, and is characterized by having an imaging data collector that collects imaging data obtained by imaging the movement of the subject's fingers, a hand tracking data generator that implements a hand tracking function that detects and tracks the position of the fingers based on the imaging data and generates hand tracking data over time from the imaging data by the hand tracking function, and a data processor that processes the hand tracking data obtained from the hand tracking data generator to generate quantitative data regarding the flexion / extension movement of the fingers and / or the opening and closing movement of two fingers associated with the movement of the finger joints.

[0014] According to the above-mentioned configuration of the present invention, the hand tracking function can generate time-dependent hand tracking data from the imaging data obtained by imaging the movements of the subject's fingers, and the hand tracking data can be processed to generate quantitative data on the flexion / extension movements of the fingers accompanying the movements of the finger joints, so that it is possible to accurately capture the movements of the joints that cannot be recognized by conventional magnetic sensor-type devices, and quantitatively evaluate the flexion / extension movements of the fingers, such as the "pinching" movement. This makes it possible to perform more detailed analysis and evaluation in the evaluation of the dexterity function of the fingers by combining the information on the inter-finger distance and the information on the movements (joint angles) of each joint of the fingers.

[0015] The measurement processing terminal having such functions may take any form. For example, the measurement processing terminal may be configured as a small terminal such as a smartphone, or may take the form of a thin tablet computer or a personal computer, or may take the form of a head mounted display (HMD, hereinafter referred to as HMD).

[0016] In the above configuration, the data processor may calculate and analyze features that lead to an evaluation of the subject's brain function. Furthermore, the data processor may evaluate the subject's brain function and cognitive function from the calculated features (for example, by comparing with data from healthy individuals). Such an evaluation can be effective as an early stage screening for dementia and can help detect dementia. Furthermore, the use of a measurement processing terminal with such a data processor is not limited to the clinical field, and it can also contribute to the evaluation of judgment in driving a car, and can be applied to brain training games, and has a wide range of applications.

[0017] In addition, in the above configuration, a movement distance measuring device that measures the distance of movement of the hand over time as the arm moves, and a gaze detector that detects the gaze of the subject's eyes may be further provided. In this case, it is preferable that the data processor processes the hand tracking data obtained from the hand tracking data generator, the distance data and time data obtained from the movement distance measuring device, and the gaze data obtained from the gaze detector to generate correlation data that quantifies the correlation between these data. According to this, for example, when moving the arm to grasp an object, the arm movement and the opening and closing movement of the fingers can be quantitatively evaluated simultaneously (the interlocking (relationship) between the finger and arm movements can be quantitatively evaluated), and further, the interlocking (relationship) between the finger and eye movements can be quantitatively evaluated. In other words, the fingers can be evaluated in synchronization with the movement of another part of the body, and therefore the upper limb motor function can be objectively, precisely, and accurately evaluated.

[0018] The correlation data generated by the data processor may include, for example, graph data showing the relationship between the distance of the hand moving from the measurement start position to the object to be grasped and time, together with the timing of opening and closing the fingers. Such correlation data allows the user to grasp the distance of the hand moving when trying to open the fingers, the distance from the measurement start position to the object, and the time required to grasp the object. The correlation data may also include graph data showing the relationship between the inter-finger distance and time. Such correlation data allows the user to grasp the timing of opening and closing the fingers (the timing of grasping the object). The correlation data may also include graph data showing the relationship between the joint angle obtained from the hand tracking data and time, and the relationship between the distance from the measurement start position and the inter-finger distance. The graph data showing the relationship between the joint angle and time allows the user to grasp the change in the joint angle over time, and the graph data showing the relationship between the distance from the measurement start position and the inter-finger distance allows the user to grasp the extent to which the hand was moved from the measurement start position when the finger opening action was performed. In addition, to enable evaluation of finger movements in synchronization with eye movements detected by eye tracking of the gaze detector, correlation data may be generated that allows, for example, the deviation in eye gaze position relative to the object to be grasped to be displayed as a scatter plot.

[0019] In the above configuration, the data processor may generate image data related to the measurement reference position and / or the measurement history. This allows the measurement conditions to be consistent between multiple tests and measurements, and allows changes in the hand (upper limb) motor function to be understood between multiple tests and measurements. Examples of image data related to the measurement reference position include image data for marking the measurement start position (such as the position where the hand is placed) on the display screen, and image data for displaying a guide on the display screen showing the outline of the hand and fingers that specifies the orientation and position of the hand at the start of measurement. Examples of image data related to the measurement history include image data for displaying past measurement results on the display screen using dotted lines, the outline of the hand, etc.

[0020] In addition, in the above configuration, a condition setter for setting measurement conditions may be further provided, and in that case, it is preferable that the data processor generates image data and / or audio data corresponding to the measurement conditions set by the condition setter. According to this, by setting the measurement conditions, for example, measurement errors (variations) can be reduced, the test and measurement environment can be standardized, and the test and measurement conditions can be made uniform (unified). As a result, problems inherent to the use of photographed data can be eliminated (for example, a standard can be set in the Z-axis direction, taking into account parameters that may vary as the distance from the camera as the imaging means to the subject changes), external factors (noise, etc.) can be reduced as much as possible, or multiple subjects can be measured under the same environment as much as possible. Therefore, for example, it is possible to accurately grasp the degree of recovery of the upper limb motor function, or to prevent variation in the test and measurement conditions between subjects. In addition, if the set conditions are visualized or audible by image or sound, the measurement conditions can be reliably recognized by the subject (or external factors that adversely affect the measurement can be eliminated), an appropriate measurement environment can be prepared, and accurate measurement results can be obtained.

[0021] Here, the measurement conditions set by the condition setter may be related to the measurement environment and the subject's field of vision, and in that case, it is preferable that the data processor generates audio data that eliminates noise in the measurement environment and / or image data that restricts the subject's field of vision. For example, the audio data that eliminates noise in the measurement environment can be audio data for outputting music from a speaker that cancels surrounding noise, and for example, the image data that restricts the subject's field of vision can be image data for inserting a virtual object between the subject's finger and surrounding objects on the display screen to hide them.

[0022] In addition, the measurement conditions set by the condition setter may be associated with the initial settings of the measurement. Examples of such initial settings include setting the measurement reference position (such as the subject's head, fingers, and gaze position) that is performed first for each measurement when performing measurements on the same subject multiple times to accurately grasp the degree of recovery of the upper limb motor function, and setting the acquisition position of the image data (for example, the camera position). In addition, such condition settings can be performed as pre-processing of the measurement process (examination), which can contribute to the standardization of the measurement conditions.

[0023] In the above configuration, the imaging data collector may have a camera that captures the movement of the fingers. Such a camera may capture the measurement environment. In the above configuration, the measurement processing terminal may further have an output interface that outputs data generated by the hand tracking data generator and / or the data processor. Such an output interface may include a display device that displays images, text, etc., and an audio output device such as a headphone or a speaker.

[0024] In addition to the above-mentioned measurement processing terminal, the present invention also provides a method and a computer program for measuring finger movements and processing the measurement results. Effect of the Invention

[0025] According to the present invention, a hand tracking function is implemented, and hand tracking data, finger movement distance data, and time data are processed to obtain correlation data that quantifies the correlation between these data. This not only makes it possible to capture the movement of the finger joints and quantitatively evaluate the finger flexion / extension motor function and / or the opening and closing motor function of two fingers, but also makes it possible to objectively, precisely, and accurately evaluate the motor function of the upper limbs. [Brief description of the drawings]

[0026] [Figure 1] 1 is a block diagram showing an example of the configuration of a measurement processing terminal as an HMD according to a first embodiment of the present invention. [Diagram 2] 2 is a schematic diagram showing a state in which a finger tapping movement is captured by a camera of the measurement processing terminal as the HMD shown in FIG. 1 worn on the head of a subject. FIG. [Diagram 3] This is a schematic diagram showing a hand tracking image in which hand landmark displays are superimposed on an image of fingers using the hand tracking function, and numerical data of joint angles obtained by processing the hand tracking data, displayed on an HMD screen. [Figure 4] 13 is a schematic diagram showing a state in which a guide display showing the outlines of fingers that define the orientation and position of the hand at the start of measurement is displayed on the HMD screen. FIG. [Diagram 5] This is a schematic diagram showing a state in which a guide display showing the outlines of the fingers that determine the orientation and position of the hand at the start of measurement, and dotted lines showing past measurement results (degree of finger lift and spread) are displayed on the HMD screen. [Figure 6] 1 is a schematic diagram showing a state in which a subject places his / her hand at a measurement start position when performing a test in which the subject grasps an object placed at a distance with his / her hand. FIG. [Figure 7] (a) is a schematic diagram showing how the HMD camera reads the position of the subject's hand when the subject places his or her hand at the measurement start position, and (b) is a schematic diagram showing how the camera constantly tracks the position of the hand even when the subject changes his or her line of sight within the capture range of the HMD camera. [Figure 8] FIG. 1 is a schematic diagram showing a subject moving his / her arm to move his / her hand towards an object placed at a distance. [Figure 9] FIG. 1 is a schematic diagram showing a subject grasping an object placed at a distance with his or her hand. [Figure 10] 1 is a graph (correlation data) showing the relationship between the distance the hand moves from the measurement start position to the object to be grasped and the time, together with the timing of opening and closing the fingers. [Figure 11] 1 is a graph (correlation data) showing the relationship between inter-finger distance and time. [Figure 12] 1 is a graph (correlation data) showing the relationship between joint angles and time obtained from hand tracking data. [Figure 13] 13 is a graph (correlation data) showing the relationship between the distance from the measurement start position and the distance between the two fingers. [Figure 14] 13 is a schematic diagram showing the state in which a gaze alignment marker is displayed on the HMD screen when measuring finger tapping movements, where (a) shows the state in which the entire image captured by the camera, including the measurement environment, is displayed as is on the HMD screen, and (b) shows the state in which an image that occludes the measurement environment except for the fingers (an image that limits the subject's field of vision) is displayed on the HMD screen. [Figure 15] The figures show images displayed on the HMD screen when a test is performed in which the subject performs finger tapping movements without looking at his or her hand. (a) is a display image in which the fingers are obscured by transparent masking, and (b) is a display image in the display state of (a), with the outlines of the fingers superimposed on the transparent masking to serve as a guide for where to place the hand. [Figure 16] 5 is a flowchart showing an example of the operation of the measurement processing terminal serving as the HMD according to the first embodiment of the present invention. [Figure 17] FIG. 11 shows an example of testing and measurement of a subject alone using a measurement processing terminal as a smartphone in accordance with a second embodiment of the present invention, in which (a) is a schematic diagram showing how the subject's face and eye movements are captured with the smartphone's in-camera, and how the subject's finger tapping movements are tracked and captured with the smartphone's out-camera, and (b) is a schematic diagram showing how the position and orientation of the hand at the time of measurement are read in advance and registered. [Figure 18] FIG. 18 is a schematic diagram showing a state in which a doctor or the like is conducting online medical examination using an inner camera in the state shown in FIG. 17(a). [Figure 19] This is a schematic diagram showing how a measurer such as a doctor tracks and photographs the movement of a subject's fingers using the rear camera of a smartphone. [Figure 20] This is a schematic diagram showing how both the subject and the measurer share examination and measurement images using a smartphone (foldable smartphone) that has two display screens that can be opened and closed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, by providing the following technologies, highly advanced technology contributes to the development of medical care and the realization of a healthy society. The realization of this measurement processing terminal contributes to "9. Build resilient infrastructure, promote inclusive and sustainable industrialization and promote innovation" of the Sustainable Development Goals (SDGs) advocated by the United Nations. In the following, the measurement and processing terminal of the present invention, which measures and processes finger movements, will be described as a head mounted display (HMD) (first embodiment) or as a smartphone (second embodiment). However, the measurement and processing terminal of the present invention may take the form of a thin tablet computer or a personal computer, or may be used in such a way that it is connected to a server via a communication means (network), and may have any structural form or usage form.

[0028] In addition, in the following embodiments, a measurement processing terminal is shown that is equipped with a camera, display, etc. and can acquire imaging data by itself and measure, process, and display the movements of fingers; however, the present invention may be embodied as a terminal or method that can measure and process the movements of fingers in cooperation with a separate imaging camera and display, or may be configured as a computer program that enables such measurement processing to be performed by a computer.

[0029] 1 to 16 show a first embodiment of the present invention in which a measurement processing terminal that measures the movements of the subject's fingers and processes the measurement results is embodied as a head mounted display (HMD) 50. Fig. 1 shows a block diagram of the configuration of such an HMD 50. 1, the HMD 50 has first and second cameras 6 and 8, a distance detection sensor 10, an optional geomagnetic sensor (gravity sensor) 25, a hand tracking data generator 26, a condition setter 24, a data processor 27, right and left eye gaze detectors 12 and 14, a display device 16, an optional operation input interface 19, a microphone 18, a speaker 20, a memory 28 containing a program 29 and information data 32, a communication interface 22, and a transmitting / receiving antenna 23, and these components are connected to each other via a bus 39, except for the transmitting / receiving antenna 23. In this case, at least the first and second cameras 6 and 8 and the distance detection sensor 10 constitute a measuring instrument that measures the movement of the subject's fingers. In addition, in this embodiment, first and second cameras 6, 8 are provided for imaging the movement of the subject's fingers, and these cameras 6, 8 constitute an imaging data collector 9 that collects imaging data obtained by imaging the movement of the subject's fingers. However, in another embodiment, cameras 6, 8 may not be provided, and imaging data obtained by imaging the movement of the subject's fingers using a camera separate from the measurement processing terminal may be input into the imaging data collector 9 via a data input interface.

[0030] The first camera 6 is an out-camera built into the HMD 50 to capture the subject's finger movements including the measurement environment (surrounding objects and scenery), and the second camera 8 is built into the HMD 50 as an in-camera that captures the subject's eyes for eye tracking by the gaze detectors 12 and 14. Both cameras 6 and 8 capture the subject and import the captured image (imaging data).

[0031] The distance detection sensor 10 constitutes a moving distance measuring device that measures the moving distance of the hand over time as the arm moves, and is a sensor that can capture the shape of an object such as a person or an object in three dimensions (or a timer that measures time may be provided separately). Examples of such sensors include LiDAR (Light Detection and Ranging), which irradiates an object with laser light such as infrared light and measures the scattered light that bounces back to analyze and detect the distance to a distant object and the state of the object, TOF (Time Of Flight) sensors that measure distance by measuring the reflection time of pulsed light irradiated on a subject for each pixel, and millimeter wave radar that emits millimeter wave radio waves and captures the reflected waves to detect the distance to the object that is reflecting the wave and the state of the object. In particular, the distance detection sensor 10 of this embodiment can detect the distance to the subject's fingers and the angle thereof, respectively, and enables measurement of each distance over time.

[0032] The right eye gaze detector 12 and the left eye gaze detector 14 detect the gaze of the subject's right eye and left eye, respectively. The process of detecting the gaze may utilize a well-known technology that is generally used as an eye tracking process. For example, in a method using corneal reflex, a technology is known in which an infrared LED (Light Emitting Diode) is irradiated onto the face, an image is taken with an infrared camera, and the position on the cornea of ​​the reflected light produced by the infrared LED irradiation (corneal reflex) is used as a reference point, and the gaze is detected based on the position of the pupil relative to the position of the corneal reflex.

[0033] The geomagnetic sensor 25 is a sensor (gravity sensor) that detects the magnetic force of the earth, and detects the direction in which the HMD 50 is facing (the angle of the subject's neck). A three-axis type geomagnetic sensor that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions is used as the geomagnetic sensor, and by capturing the geomagnetic change in response to the movement of the HMD 50, it is also possible to detect the movement of the HMD 50 (the angle of the subject's neck).

[0034] The condition setter 24 is for setting the measurement conditions, and is configured by a user interface that allows the selection of an examination mode such as an upper limb motor function or a finger tapping motor function, and the selection of measurement conditions prepared for each examination mode, and the measurement conditions can be set by displaying this user interface on the display screen of the HMD 50 and inputting through an input means such as gesture operation, voice input, or a keyboard, key buttons, touch keys, etc. Measurement conditions for the finger tapping motor function prepared as a user interface include, for example, selection of a measurement mode such as both hands simultaneously, both hands alternately, one hand (right hand) only, or one hand (left hand) only, selection of whether to show the subject's fingers during measurement (on / off of a transparent masking function for the fingers), and selection of whether to suppress noise related to the measurement environment entering through the eyes and ears (factors that prevent the subject from concentrating on the measurement) (on / off of a function for hiding surrounding information by inserting a virtual object between the fingers and surrounding objects, for example). As an example of a measurement condition provided as a separate user interface, it may be possible to set a function that displays past measurement results and target guidelines using finger outlines or dotted lines, for example, so that changes in finger (upper limb) motor function (in the case of rehabilitation training, the rehabilitation effects) can be understood.

[0035] The hand tracking data generator 26 also implements a hand tracking function that detects and tracks the positions of the fingers based on the imaging data acquired by the camera 6, and generates time-dependent hand tracking data from the imaging data by the hand tracking function. As the hand tracking (skeleton detection) function, for example, the open source machine learning tool "MediaPipe" provided by Google in the United States may be used.

[0036] The data processor 27 processes the hand tracking data obtained from the hand tracking data generator 26 to generate quantitative data on the flexion / extension movement of the fingers and / or the opening / closing movement of the two fingers accompanying the movement of the finger joints, and also processes the hand tracking data obtained from the hand tracking data generator 26, the distance data and time data obtained from the distance detection sensor 10, and the gaze data obtained from the gaze detectors 12 and 14 to generate correlation data that quantifies the correlation between these data. The data processor 27 also generates image data related to the measurement reference position and / or the measurement history, and also generates image data and / or audio data corresponding to the measurement conditions set by the condition setter 24. In particular, when the measurement conditions are related to the measurement environment and the subject's field of vision, the data processor 27 generates audio data that eliminates noise in the measurement environment and / or image data that limits the subject's field of vision. In this embodiment, the data processor 27 also constitutes the controller of the HMD 50, and is composed of a CPU etc., and executes programs 29 such as an operating system (OS) 30 and various operation control applications 31 stored in memory 28, thereby performing operation control processing for the entire HMD 50 and controlling the startup operations of various applications.

[0037] The memory 28 is composed of a flash memory or the like, and stores programs 29 such as an operating system 30 and an application 31 for controlling various processes such as images, sounds, documents, display, and measurement. The memory 28 also stores information data 32 such as base data 33 required for basic operations by the operating system 30 and file data 34 used by the various applications 31. For example, an image processing application is started, an image is taken with a camera, and the captured file data is saved and stored. The processing in the data processor 27 may be stored as one application A, and the measurement process of the finger movements and the calculation and analysis of various feature amounts may be performed by starting the application A. Also, an external server device with high computing performance and large capacity may receive the measurement results measured from the information processing terminal and calculate and analyze the feature amounts.

[0038] The display device 16 is an output interface that outputs data generated by the hand tracking data generator 26 and / or the data processor 27, and can display the results of processing by the data processor 27. In the case of an optical transmission type HMD, the display device 16 is composed of a projection unit that projects various information such as playback information by a running application and notification information to the subject, and a transparent half mirror that forms an image of the projected various information in front of the subject's eyes. In the case of a video transmission type HMD, the display device 16 is composed of a display such as a liquid crystal panel that displays a real space object in front of the subject's eyes photographed by the first camera 6 together with various information. This allows the subject to view and listen to not only the image in the field of view in front of the subject's eyes, but also other image information.

[0039] In the second embodiment described later, since the measurement processing terminal is a smartphone, the display device 16 is composed of an LCD panel or the like, and in addition to displaying images and videos, it is possible to display notification information for the subject, such as the remaining battery capacity, various alarms, and the time, as well as icons of apps to be launched on the display screen.

[0040] The operation input interface 19 of the HMD 50 often uses gesture operation or voice input, but may also use input means such as a keyboard, key buttons, or touch keys, and is used to set and input information that the subject wants to input. In the second embodiment described later, since the measurement processing terminal is a smartphone, the operation input interface 19 may be provided on the terminal itself, but in this embodiment, the operation input interface 19 may be provided in a position or form that makes it easy for the subject to perform input operations in the HMD 50, or may be separated from the main body of the HMD 50 and connected by wire or wirelessly. In addition, an input operation screen may be displayed on the display screen of the display device 16, and input operation information may be captured based on the position on the input operation screen to which the gaze detected by the right eye gaze detector 12 and the left eye gaze detector 14 is directed, or a pointer may be displayed on the input operation screen and the pointer may be operated by the operation input interface 19 to capture the input operation information. In addition, the subject may utter a voice indicating an input operation, and the sound may be collected by the microphone 18 to capture the input operation information.

[0041] The microphone 18 can also constitute an output interface that outputs audio data generated by the data processor 27, and collects external sounds and the user's own voice. The speaker 20 outputs sounds to the outside and informs the user of sounds such as notification information and music. The speaker 20 may also be used to give the subject instructions regarding the measurement of finger movements by voice.

[0042] The communication interface 22 is a communication interface that performs wireless communication with a server device or the like located in another place by short-distance wireless communication, wireless LAN, or base station communication. During wireless communication, the communication interface transmits and receives measurement data, analyzed and calculated features, and the like to and from the server device or the like via a transmission / reception antenna 23. The short-distance wireless communication is performed using, for example, an electronic tag, but is not limited to this, and may be performed using a wireless LAN such as Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or Wi-Fi (registered trademark) as long as it is capable of at least wireless communication when it is close to another information terminal. In addition, long-distance wireless communication such as W-CDMA (Wideband Code Division Multiple Access) or GSM (registered trademark) may be used as the base station communication. It is also possible to detect the positional relationship and orientation between terminals using an ultra-wideband wireless system (Ultra Wide Band: UWB). Although not shown, the communication interface 22 may use other methods such as communication by optical communication and sound waves as a means of wireless communication. In that case, instead of the transmitting / receiving antenna 23, a light emitting / receiving unit and a sound wave output / sound wave input interface are used, respectively.

[0043] In this embodiment, the HMD 50 has each of the components described above individually, but it may also have a functional unit that integrates at least some or all of these components. In short, as long as the functionality of each of these components is ensured, any configuration is acceptable.

[0044] Next, the measurement processing operation of the HMD 50 that measures and processes the movements of the fingers will be described based on FIGS. 1 to 15 and with reference to the flowchart shown in FIG. 2 shows a state in which a finger tapping movement performed by a subject 60 is captured by a first camera 6 of an HMD 50 attached to a head 62 of the subject 60. A moving image 70 of the fingers of the subject 60 captured by the first camera 6 is displayed on the screen of a display device 16 of the HMD 50.

[0045] 3 shows a hand tracking image 75 in which a hand landmark display 72 is superimposed on a moving image 70 of the hand and fingers by the hand tracking function described above, and various information related to measurement obtained by the data processor 27 processing the hand tracking data generated by the hand tracking data generator 26, such as numerical data 73 of joint angles, displayed on the screen 16a of the display device 16 of the HMD 50. Of course, the data related to the hand landmarks is essentially used in the data processing in the data processor 27, and therefore does not have to be displayed to the subject (the hand landmark display 72 does not have to be displayed on the screen 16a).

[0046] FIG. 4 shows a state in which a guide display (guide contour) 76 showing the outline of the hand and fingers that specifies the orientation and position of the hand at the start of measurement is displayed on the screen 16a of the display device 16 of the HMD 50 as image data related to the measurement reference position together with a hand tracking image 75. This guide display 76 is obtained by reading and registering (storing in the memory 28) the position and orientation of the hand at the time of measurement in advance via the first camera 6, and is read from the memory 28 for each measurement and displayed on the screen 16a at the position at the time of reading. Such a guide display 76 can not only form image data related to the measurement reference position, but also form image data for initial setting that is performed first for each measurement when performing multiple measurements on the same subject to accurately grasp the degree of recovery of the upper limb motor function, so that the measurement conditions can be aligned between multiple tests and measurements. Of course, instead of the outline of the fingers, an image of the hand through a transparent film may be displayed. Note that the image data related to the measurement reference position is generated by the data processor 27.

[0047] FIG. 5 shows a state in which a dotted line 79 showing past measurement results (how the fingers are raised and spread) is displayed on the screen 16a of the display device 16 of the HMD 50 as image data related to the measurement history, together with a guide display 76 showing the outline of the fingers that defines the orientation and position of the hand at the start of measurement. Of course, instead of the dotted line 79, the outline of the fingers showing how the fingers were raised and spread in the past may be displayed. Such image data related to the measurement history allows changes in the motor function of the fingers (upper limbs) between multiple tests and measurements (in the case of rehabilitation training, the rehabilitation effect). The image data related to the measurement history is generated by the data processor 27. In addition, when the HMD50 is shared by multiple people, unique identification information such as fingerprints or palm lines can be registered in advance, and the contour information of the subject's fingers can be read and displayed by comparing the information before measurement.

[0048] 6 shows a state where a subject 60 places his / her hand 63 at a measurement start position when performing a test in which the subject grasps an object 80 placed at a distance with his / her hand. Such a measurement start position is displayed as a marking 83 on the screen 16a of the display device 16 of the HMD 50 (for example, on a desk 93 shown in an image capturing the measurement environment) as image data related to the measurement reference position. At the start of such a test, for example, it is conditioned that the hand 63 is placed at the position of such marking 83 (step S1 in FIG. 16). Note that the image data related to the measurement reference position is generated by the data processor 27.

[0049] FIG. 7(a) shows how the first camera 6 of the HMD 50 reads the position of the hand 63 in a state where the subject 60 places his / her hand 63 on the marking 83. At the start of the test, such a starting position is also read (step S2 in FIG. 16). FIG. 7(b) is a schematic diagram showing how the first camera 6 of the HMD 50 constantly tracks the position of the hand 63 even when the subject 60 changes his / her line of sight L within the capture range of the first camera 6. In this sense, it is preferable that the first camera 6 can capture the position of the hand 63 over a wide range.

[0050] Furthermore, when the measurement conditions are set by the condition setter 24 at the start of the test (condition setting step S3 in FIG. 16), the data processor 27 generates image data and / or voice data corresponding to the measurement conditions (step S4 in FIG. 16). By setting the measurement conditions, for example, it is possible to reduce measurement errors (variations), standardize the test and measurement environment, and make the test and measurement conditions uniform (unify). As a result, it is possible to solve problems inherent to the use of photographed data (for example, to consider parameters that may vary as the distance from the first camera 6 to the object 80 changes, and to make the standard uniform in the Z-axis direction), to reduce external factors (noise, etc.) as much as possible, or to allow multiple subjects to measure under the same environment as much as possible. Therefore, for example, it is possible to accurately grasp the degree of recovery of the upper limb motor function, or to prevent variations in the test and measurement conditions between subjects. Furthermore, if the set conditions are visualized or audible using images or audio, the measurement conditions can be reliably recognized by the subject 60 (or external factors that adversely affect the measurement can be eliminated), an appropriate measurement environment can be created, and accurate measurement results can be obtained.

[0051] In this embodiment, the measurement conditions set by the condition setter 24 are associated with the measurement environment and the field of view of the subject 60, and the data processor 27 generates audio data for eliminating noise in the measurement environment and / or image data for restricting the field of view of the subject 60. For example, the data processor 27 generates audio data for outputting music for canceling ambient noise from the speaker 20 as audio data for eliminating noise in the measurement environment in order to suppress unnecessary information entering through the ears and eyes, and generates image data for inserting and hiding a virtual object between the fingers of the subject 60 and surrounding objects on the screen 16a as image data for restricting the field of view of the subject 60. That is, the HMD 50 is used to play images and music that allow the subject 60 to perform the measurement in a relaxed state.

[0052] An example of image data generation related to such measurement condition setting is shown in Fig. 14 and Fig. 15. Fig. 14 shows a state in which a text display 87 and a gaze alignment marker 85 are displayed on the screen 16a of the display device 16 of the HMD 50 when a subject 60 performs a finger tapping movement with the fingers of both hands 63, 63. In this case, Fig. 14(a) shows a state in which the entire image captured by the first camera 6 including the measurement environment is displayed as it is on the screen 16a, and Fig. 14(b) shows a state in which an image in which the measurement environment except for the hand 63 is blocked (masking 91 that limits the subject's field of view) is displayed on the screen 16a. The masking 91 is image data that limits the field of view of the subject 60, and the gaze alignment marker 85 is image data corresponding to the measurement condition for concentrating the subject on the measurement, and can also function as image data related to the measurement reference position or image data related to the initial setting.

[0053] Also, Figure 15 shows images displayed on the screen 16a when a test is performed in which the subject 60 performs finger tapping movements without looking at his or her hand 63. (a) of Figure 15 is a displayed image in which the hand 63 of the subject 60 is shielded by a transparent masking 89, and (b) of Figure 15 is a displayed image in which the outlines 76 of the fingers, which serve as a guide for where to place the hand 63, are superimposed on the transparent masking 89 in the display state of (a) of Figure 15.

[0054] In this way, in this embodiment, the hand part can be switched between visible and invisible depending on the measurement purpose. That is, in the case of a measurement in which the finger movement is to be shown, only the hand 63 part is displayed in the image captured by the first camera 6 as shown in Fig. 14, and in the case of a measurement in which the finger movement is not to be shown, the hand 63 part is masked as shown in Fig. 15 (or only the guide (outline) display 76 for positioning is performed). By preventing the finger movement from being shown to the subject 60, the influence of visual information on the measurement of the finger tapping movement can be suppressed, and the measurement conditions can be made uniform (standardized).

[0055] 14, when the gaze alignment marker 85 is displayed, it is preferable to perform eye tracking using the second camera 8 as an in-camera and the gaze detectors 12, 14, and to prompt the subject 60 to repeat the measurement when it is detected that the subject's gaze has deviated from the marker 85 during measurement (for example, by playing a voice prompting the subject to repeat the measurement from the speaker 20, or by displaying text prompting the subject to repeat the measurement on the screen 16a). Alternatively, the angle of the neck of the subject 60 wearing the HMD 50 may be measured using the geomagnetic sensor 25, and the subject 60 may be prompted to face forward rather than toward the hand (without tilting the neck down).

[0056] FIG. 8 shows a state where the subject 60 moves his / her arm 64 to move his / her hand 63 to an object 80 placed at a distant position. When the measurement is started in this way (step S5 in FIG. 16), the imaging data collector 9 (first camera 6) captures the image of the movement of the subject's 60's fingers to obtain imaging data, and the hand tracking data generator 26 generates time-dependent hand tracking data from the imaging data by the hand tracking function (step S6 in FIG. 16 (imaging step, imaging data acquisition step, and hand tracking data generation step)). At the same time, the distance over time that the hand 63 moves in accordance with the movement of the subject's 60's arm 64 is measured by the distance detection sensor 10 (the time it takes to move the hand 63 to grasp the object 80 is also calculated), and the line of sight of the subject's 60's eyes is detected by the second camera 8 and line of sight detectors 12, 14 (detecting where the subject 60 is looking at what timing...step S7 in FIG. 16 (movement distance measurement step and line of sight detection step)). That is, as described above, the hand 63 placed on the marking 83 is read and recognized via the first camera 6, and then the distance traveled by the hand 63 over time from the read-in starting position in association with the movement of the arm 64 is calculated by the data processor 27 via the distance detection sensor 10.

[0057] Then, when the subject 60 grasps the object 80 placed at a distance with his / her hand 63 as shown in Fig. 9 (step 8 in Fig. 16), the data processor 27 processes the hand tracking data obtained from the hand tracking data generator 26 to generate quantitative data on the flexion / extension movement of the fingers and / or the opening / closing movement of the two fingers accompanying the movement of the knuckles, and processes the hand tracking data obtained from the hand tracking data generator 26, the distance data and time data obtained from the distance detection sensor 10, and the gaze data obtained from the gaze detectors 12 and 14 via the second camera 8 to generate correlation data that quantifies the correlation between these data (data processing step S9). With such correlation data, the movement of the arm 64 and the opening / closing movement of the fingers (this opening / closing movement can be grasped by the hand tracking function) can be evaluated in synchronization (the timing of the opening and closing of the fingers can be detected and evaluated in the series of movements from the arm 64 starting to move to grasp the object 80).

[0058] An example of such correlation data is shown in FIG. 10 to FIG. 13. FIG. 10 is a graph showing the relationship between the moving distance of the hand from the measurement start position to the object to be grasped and time, together with the timing of opening and closing the fingers. Such correlation data makes it possible to know the moving distance of the hand when trying to open the fingers, the distance from the measurement start position to the object, and the time required to grasp the object. FIG. 11 is a graph showing the relationship between the distance between two fingers and time. Such correlation data makes it possible to know the timing of opening and closing the fingers (the timing of grasping the object). FIG. 12 is a graph showing the relationship between the joint angle obtained from the hand tracking data and time. Such correlation data makes it possible to know the change over time in the joint angle. FIG. 13 is a graph showing the relationship between the distance from the measurement start position and the distance between two fingers. Such correlation data makes it possible to know how far the hand was moved from the measurement start position when the finger opening action was performed. In order to enable evaluation of finger movements in synchronization with eye movements detected by eye tracking of the gaze detectors 12 and 14, correlation data may be generated that allows the deviation of the gaze position of the eye relative to the object to be grasped to be displayed as a scatter plot, for example.

[0059] Such correlation data, together with other data generated by the hand tracking data generator 26 and / or the data processor 27, can be displayed on the screen 16a (output step S10 in FIG. 16).

[0060] Next, a second embodiment of the present invention will be described with reference to Fig. 17 to Fig. 20. In this embodiment, the measurement processing terminal is configured as a smartphone 100. Note that the basic configuration and operation are the same as those already described with reference to Fig. 1 and Fig. 16 etc. 17 and 18 show an example of use in which the smartphone 100 is placed between the face 69 and hand 63 of the subject 60, and the subject performs the test and measurement by himself / herself. Here, (a) of FIG. 17 shows how the second camera 8 (in-camera) of the smartphone 100 captures the face and eye movements of the subject 60, and the first camera 6 (out-camera) of the smartphone 100 tracks and captures the finger tapping movement (and / or the measurement environment) of the subject 60. Note that it is preferable to fix the smartphone 100 in place, rather than holding it in the hands, in order to prevent shaking and to measure both hands simultaneously. In addition, various information related to the measurement, including the user interface for hand tracking (as already described above, the same reference numerals are given in the figures and the description is omitted... the same applies to FIGS. 18 to 20) is displayed on the screen 16a of the display device 16 of the smartphone 100. In addition, the face and eye movements of the subject 60 captured by the second camera 8 are displayed as an inset image 110 on the screen 16a.

[0061] 17(b) shows how the position and orientation of the hand 63 at the time of measurement are read and registered in advance. The registered information is read out before measurement, and a hand outline guide 76 is displayed in the position at the time of reading. In addition, the displays described above for the HMD, such as the hand outline display and gaze marker, are also displayed in the same way on the smartphone 100.

[0062] Fig. 18 shows a state in which a doctor or the like is conducting an online medical examination using the second camera 8 in the state shown in Fig. 17(a). In this way, it is possible to perform measurements while talking with the doctor who is displayed as an inset image 112 on the screen 16a using FaceTime (registered trademark) or the like.

[0063] 19 and 20 show an example of use in which the smartphone 100 is placed behind the hand 63 of the subject 60 and a measurer 120 such as a doctor measures the finger tapping movement of the subject 60. Here, FIG. 19 shows a state in which the measurer 120 such as a doctor holds the smartphone 100 in his hand and tracks and captures the movement of the subject 60's fingers together with the upper body from the opposite side with the first camera 6 of the smartphone 100. The measurer 120 can display and check various information related to the measurement, including a user interface for hand tracking, on the screen 16a of the smartphone 100. The subject 60 performs a finger tapping movement toward the smartphone 100.

[0064] FIG. 20 shows how both the subject 60 and the measurer 120 share test and measurement images using a smartphone (foldable smartphone) 100A having two display screens 16a, 16a' that can be opened and closed. That is, here, an image captured by a camera on the subject 60 side is also displayed on the screen 16a' on the measurer 120 side and shared. Specifically, the subject 60 side displays the image captured by the camera, and the measurer 120 side displays the image with tracking information or the like added. For example, only on the screen on the subject 60 side, the hand 63 is masked by a masking 89, or a gaze marker 85 or a contour 76 for positioning is displayed. Also, images that have been processed differently (with different information added) may be displayed for the subject 60 and the measurer 120, such as an image captured by a camera and with a hand landmark display 72 superimposed on only the screen 16a' on the measurer 120 side. Also, only the contour may be displayed on the subject 60 side as a guide for rehabilitation training.

[0065] As described above, according to the first and second embodiments, the hand tracking function can generate time-dependent hand tracking data from imaging data obtained by imaging the movements of the fingers of the subject 60, and the hand tracking data can be processed to generate quantitative data on the flexion / extension movements of the fingers and / or the opening and closing movements of the two fingers accompanying the movements of the finger joints, making it possible to accurately capture joint movements that cannot be recognized by conventional magnetic sensor-type devices, and quantitatively evaluate the flexion / extension movements of the fingers, such as "pinching" movements. Unlike the case of conventional magnetic sensor-type devices, there is no need to attach sensors to the fingertips.

[0066] According to the first and second embodiments, the hand tracking data obtained from the hand tracking data generator 26, the distance data and time data obtained from the distance detection sensor 10, and the gaze data obtained from the gaze detectors 12 and 14 can be processed to generate correlation data that quantifies the correlation between these data, so that, for example, when moving an arm to grasp an object, the arm movement and the opening and closing movement of the fingers can be simultaneously and quantitatively evaluated (the linkage (relationship) between the finger and arm movements can be quantitatively evaluated), and further, the linkage (relationship) between the finger and eye movements can be quantitatively evaluated. In other words, the fingers can be evaluated in synchronization with the movement of another part of the body, and therefore the upper limb motor function can be objectively, minutely, accurately, and precisely evaluated.

[0067] Although the embodiment of the present invention has been described above with reference to the drawings, the present invention is not limited to the above-described embodiment and can include various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to the embodiment having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0068] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as a program, table, file, etc. that realizes each function may be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, DVD, etc., or may be stored in a device on a communication network.

[0069] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0070] 6,8 Camera 9 Imaging Data Collector 10 Distance detection sensor (travel distance measuring device) 12,14 Line of Sight Detector 16 Display device (output interface) 20 Speaker (output interface) 24 Condition Setter 26 Hand Tracking Data Generator 27 Data Processor 50 HMD (measurement processing terminal) 100,100A Smartphone (measurement processing terminal)

Claims

1. A measurement processing terminal that measures the movement of a subject's fingers and processes the measurement results, an imaging data collector that captures imaging data of the subject's finger movements; a hand tracking data generator that implements a hand tracking function that detects and tracks the position of a hand or fingers based on the imaging data and generates time-dependent hand tracking data from the imaging data by the hand tracking function; A movement distance measuring device that measures the distance the hand moves over time in response to arm movement; a data processor that processes the hand tracking data obtained from the hand tracking data generator and the distance data and time data obtained from the movement distance measuring device to generate quantitative data on finger flexion / extension movement and / or opening / closing movement of two fingers accompanying the movement of the arm and finger joints, and generates correlation data that quantifies the correlation between these data, the correlation data enabling evaluation of dexterous motor function of the fingers by combining information on the distance between the two fingers and information on the movement of each joint of the fingers; A measurement processing terminal comprising:

2. A gaze detector for detecting a gaze of the subject's eye is further provided, the data processor processes the hand tracking data obtained from the hand tracking data generator, the distance data and time data obtained from the movement distance measuring device, and the gaze data obtained from the gaze detector, to generate the correlation data quantifying the correlation between these data.

2. The measurement processing terminal according to claim 1.

3. 3. The measurement processing terminal according to claim 1, wherein the data processor generates image data related to the measurement reference position and / or the measurement history.

4. A condition setting device for setting measurement conditions is further provided, The data processor generates image data and / or audio data corresponding to the measurement conditions set by the condition setter.

4. The measurement processing terminal according to claim 1, wherein the measurement processing terminal is a terminal for measuring a temperature and a humidity of the measurement object.

5. 5. The measurement processing terminal according to claim 4, wherein the measurement conditions are associated with a measurement environment and a field of view of a subject.

6. 6. The measurement processing terminal according to claim 5, wherein the data processor generates voice data for eliminating noise in the measurement environment and / or image data for restricting the subject's field of vision.

7. 5. The measurement processing terminal according to claim 4, wherein the measurement conditions are associated with initial measurement settings.

8. 8. The measurement processing terminal according to claim 1, wherein the imaging data collector has a camera for imaging the movement of fingers.

9. 9. The measurement processing terminal according to claim 1, further comprising an output interface for outputting data generated by the hand tracking data generator and / or the data processor.

10. A measurement processing method for measuring the movement of a subject's fingers and processing the measurement results, comprising: An imaging data acquisition step of acquiring imaging data obtained by imaging the movement of the subject's fingers; a hand tracking data generating step of generating time-dependent hand tracking data from the imaging data by a hand tracking function that detects and tracks the position of the hand and fingers based on the imaging data; a movement distance measuring step of measuring a movement distance over time of the hand moving in accordance with the arm movement; a data processing step of processing the hand tracking data obtained from the hand tracking data generating step and the distance data and time data obtained from the movement distance measuring step to generate quantitative data on finger flexion / extension movement and / or opening / closing movement of two fingers accompanying the movement of the arm and finger joints, and generating correlation data that quantifies the correlation between these data, the correlation data enabling evaluation of dexterous motor function of the fingers by combining information on the distance between the two fingers and information on the movement of each joint of the fingers; A measurement processing method comprising:

11. The method further includes a gaze detection step of detecting a gaze of the subject's eye, The data processing step processes the hand tracking data obtained from the hand tracking data generation step, the distance data and the time data obtained from the movement distance measurement step, and the gaze data obtained from the gaze detection step, to generate the correlation data that quantifies the correlation between these data. The measurement processing method according to claim 10 .

12. 12. The measurement processing method according to claim 10, wherein the data processing step generates image data related to the measurement reference position and / or the measurement history.

13. The method further includes a condition setting step for setting measurement conditions, The data processing step generates image data and / or audio data corresponding to the measurement conditions set in the condition setting step. The measurement processing method according to any one of claims 10 to 12.

14. The measurement processing method according to claim 13 , wherein the measurement conditions are related to a measurement environment and a field of view of a subject.

15. 15. The measurement processing method according to claim 14, wherein said data processing step generates voice data for eliminating noise in the measurement environment and / or image data for restricting the subject's field of view.

16. The measurement processing method according to claim 13 , wherein the measurement conditions are associated with initial measurement settings.

17. 17. The measurement processing method according to claim 10, wherein the imaging data acquisition step includes an imaging step of imaging a movement of a finger.

18. The measurement processing method according to any one of claims 10 to 17, further comprising an output step of outputting data generated by the hand tracking data generating step and / or the data processing step.

19. A computer program for measuring the movement of a subject's fingers and processing the measurement results, An imaging data acquisition step of acquiring imaging data obtained by imaging the movement of the subject's fingers; a hand tracking data generating step of generating time-dependent hand tracking data from the imaging data by a hand tracking function that detects and tracks the position of the hand and fingers based on the imaging data; a movement distance measuring step of measuring a movement distance over time of the hand moving in accordance with the arm movement; a data processing step of processing the hand tracking data obtained from the hand tracking data generating step and the distance data and time data obtained from the movement distance measuring step to generate quantitative data on finger flexion / extension movement and / or opening / closing movement of two fingers accompanying the movement of the arm and finger joints, and generating correlation data that quantifies the correlation between these data, the correlation data enabling evaluation of dexterous motor function of the fingers by combining information on the distance between the two fingers and information on the movement of each joint of the fingers; A computer program characterized by causing a computer to execute the above.

20. A gaze detection step of detecting a gaze of the subject's eyes is further performed by the computer. The data processing step processes the hand tracking data obtained from the hand tracking data generation step, the distance data and the time data obtained from the movement distance measurement step, and the gaze data obtained from the gaze detection step, to generate the correlation data that quantifies the correlation between these data.

20. A computer program product as claimed in claim 19.

21. 21. The computer program product according to claim 19, wherein the data processing step generates image data related to the measurement reference position and / or the measurement history.

22. further causing the computer to execute a condition setting step of setting measurement conditions; The data processing step generates image data and / or audio data corresponding to the measurement conditions set in the condition setting step.

22. A computer program according to any one of claims 19 to 21.

23. 23. The computer program product of claim 22, wherein the measurement conditions are related to a measurement environment and a subject's field of view.

24. 24. The computer program according to claim 23, wherein the data processing step generates audio data that eliminates noise in the measurement environment and / or image data that limits the subject's field of view.

25. 23. The computer program product of claim 22, wherein the measurement conditions are associated with initial measurement settings.

26. 26. The computer program product according to claim 19, wherein the imaging data acquisition step includes an imaging step of imaging a movement of a finger.

27. 27. A computer program product according to any one of claims 19 to 26, further comprising causing a computer to perform an output step of outputting data generated by the hand tracking data generating step and / or the data processing step.

Images