Operation state monitoring system, operation state monitoring method, and operation state monitoring program product

Abstract

The present disclosure provides an action state monitoring system, an action state monitoring method, and an action state monitoring program product that make it possible to easily understand the action state at a target part of the body of a subject. A motion state monitoring system according to the present disclosure monitors a motion state at a target part of a subject's body, and is provided with: a calculation processing unit that generates a calculation result indicating the motion state on the basis of a detection result from a sensor that detects the motion state; an image processing unit that performs image processing of at least one of a captured image in which the operating state has been captured and a rendered image in which the operating state has been rendered; and a display unit that displays the image subjected to image processing in synchronization with the calculation result.

Description

Technical Field

[0001] The present disclosure relates to an action state monitoring system, an action state monitoring method and an action state monitoring program. Background Art

[0002] Japanese Patent Application Laid-Open No. 2022-034450 discloses an operation state monitoring system that monitors the operation state of a target part of a subject's body.

[0003] The display unit in the operation state monitoring system of Japanese Patent Application Laid-Open No. 2022-034450 displays an icon indicating the position of a sensor that detects an operation state and a line graph that indicates the operation state detected by the sensor as an angle from a predetermined position. Summary of the invention

[0004] Only by the angle display based on the line graph, it is difficult to understand the motion state at the target part of the subject's body when reviewing the data. A technology that makes it easy to understand the motion state at the target part of the subject is desired.

[0005] The present disclosure has been made in view of the above background, and an object of the present disclosure is to provide an operation state monitoring system, an operation state monitoring method, and an operation state monitoring program that can easily understand the operation state of a target part of a subject's body when reviewing data.

[0006] An action state monitoring system according to an embodiment of the present disclosure monitors the action state of an object part of a body of a subject, and comprises: an operation processing unit that generates an operation result representing the action state based on the detection result of a sensor that detects the action state; an image processing unit that performs image processing on at least one of a captured image capturing the action state and a drawn image drawing the action state; and a display unit that displays the image processed in synchronization with the operation result. The operation processing unit may also perform operation processing using a learned model generated by machine learning using past detection results of the sensor.

[0007] In the above-mentioned action state monitoring system, the following method can also be adopted, that is, the image processing unit generates the drawn image of the 3D model drawn with the action state or the drawn image of the avatar drawn with the action state, and the display unit displays the generated 3D model or the drawn image of the avatar.

[0008] In the above-mentioned action state monitoring system, the image processing unit may generate a trajectory of the object part in the rendered image of the 3D model or the avatar, and the display unit may display the rendered image including the generated trajectory.

[0009] In the above-mentioned operation state monitoring system, the image processing unit may generate a trajectory of the target part in the captured image, and the display unit may display the captured image including the generated trajectory.

[0010] In the above-mentioned action state monitoring system, the following method can also be adopted, that is, the image processing unit generates a trajectory of the object part in the drawn image of the 3D model or the avatar that draws the action state captured in the captured image, and the display unit displays the drawn image including the generated trajectory.

[0011] In the above-mentioned action state monitoring system, the following method can also be adopted, that is, the image processing unit generates a trajectory of the object part in the drawn image of the 3D model or the avatar that draws the action state that has been previously generated as a reference, and the display unit displays the drawn image including the generated trajectory.

[0012] In the above-mentioned action state monitoring system, the following method can also be adopted, that is, the image processing unit generates a first trajectory of the object part in the drawn image of the 3D model of the action state captured in the captured image or the avatar, and generates a second trajectory of the object part in the drawn image of the 3D model of the action state or the avatar that is previously generated as a reference, and generates a Lissajous figure that synthesizes the first trajectory and the second trajectory, and the display unit displays the generated Lissajous figure.

[0013] In the above-mentioned operation state monitoring system, the display unit may display the drawn image of the 3D model or the avatar by enlarging or reducing it.

[0014] In the above-mentioned action state monitoring system, the following method can also be adopted, that is, the image processing unit generates a graphic representing the movable area of ​​the object part in the drawn image of the 3D model or the avatar, and the display unit displays the generated graphic representing the movable area.

[0015] In the above-mentioned motion state monitoring system, the display unit may further display at least one of vital sign data and walking speed.

[0016] In the above-mentioned operation status monitoring system, the following mode may be adopted, that is, the display unit displays a plurality of the images at different times.

[0017] An action state monitoring method involved in one embodiment of the present invention uses an action state monitoring system to monitor the action state of an object part of a subject's body, wherein the action state monitoring system comprises: an operation processing unit, which generates an operation result representing the action state based on a detection result of a sensor that detects the action state; an image processing unit, which performs image processing on at least one of a captured image of the action state and a drawn image that draws the action state; and a display unit, which displays the image processed synchronously with the operation result. The action state monitoring method comprises: a step of generating the operation result representing the action state based on the detection result; a step of performing image processing on the image; and a step of displaying the image processed synchronously with the operation result.

[0018] An action state monitoring program involved in one embodiment of the present invention causes a computer contained in an action state monitoring system to execute processing for monitoring the action state at an object part of a subject's body, wherein the action state monitoring system comprises: an operation processing unit, which generates an operation result representing the action state based on a detection result of a sensor that detects the action state; an image processing unit, which performs image processing on at least one of a captured image of the action state and a drawn image that draws the action state; and a display unit, which displays the image processed synchronously with the operation result, wherein the action state monitoring program causes the computer to execute the following steps, namely: a step of generating the operation result representing the action state based on the detection result; a step of performing image processing on the image; and a step of displaying the image processed synchronously with the operation result.

[0019] According to the present disclosure, it is possible to provide an operation state monitoring system, an operation state monitoring method, and an operation state monitoring program that can easily understand the operation state of a target part of a subject's body when reviewing data.

[0020] The above and other objects, features and advantages of the present disclosure will be more fully understood from the detailed description and accompanying drawings given below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The following is a diagram illustrating a configuration of an operation state monitoring system according to the first embodiment.

[0022] Figure 2This is a configuration diagram illustrating how a sensor of a measuring instrument is attached in the operation state monitoring system according to the first embodiment.

[0023] Figure 3 This is a configuration diagram illustrating how a sensor of a measuring instrument is attached in the operation state monitoring system according to the first embodiment.

[0024] Figure 4 This is a block diagram illustrating a measuring instrument and an operation state monitoring device in the operation state monitoring system according to the first embodiment.

[0025] Figure 5 This is a flowchart illustrating a measurement method in the operation state monitoring method using the operation state monitoring system according to the first embodiment.

[0026] Figure 6 This is a flowchart illustrating a display method in the operation state monitoring method using the operation state monitoring system according to the first embodiment.

[0027] Figure 7 This is a diagram illustrating the display on the display unit in the operation status monitoring system according to the first embodiment.

[0028] Figure 8 This is a diagram illustrating the display on the display unit in the operation status monitoring system according to the first embodiment.

[0029] Fig. 9 This is a diagram illustrating the display on the display unit in the operation status monitoring system according to the first embodiment.

[0030] Fig.10 This is a diagram illustrating the display on the display unit in the operation status monitoring system according to the first embodiment.

[0031] Fig.11 This is a diagram illustrating the display on the display unit in the operation status monitoring system according to the first embodiment.

[0032] Fig.12 This is a diagram illustrating the display on the display unit in the operation status monitoring system according to the first embodiment.

[0033] Fig.13 The following is a schematic configuration diagram illustrating an operation status monitoring device including a computer according to an embodiment. DETAILED DESCRIPTION

[0034] Hereinafter, the present disclosure will be described by way of the embodiments, but the present disclosure is not limited to the following embodiments. In addition, not all of the structures described in the embodiments are necessary technical means for solving the problems. For the purpose of clear description, the following descriptions and drawings are appropriately omitted and simplified. In each of the drawings, the same symbols are marked for the same elements, and repeated descriptions are omitted as needed.

[0035] <Implementation method 1>

[0036] The operation state monitoring system 1 according to the first embodiment will be described. Figure 1 The structural diagram of the motion state monitoring system involved in the embodiment 1 is illustrated. The motion state monitoring system 1 monitors the motion state of the body part of the subject P. In addition, based on the monitoring result of the motion of the subject P, the motion state monitoring system 1 implements support to make the motion of the subject P close to the desired motion. Specifically, the motion state monitoring system 1 measures the motor function of the subject P including the rehabilitation trainee and the elderly. Moreover, the motion state monitoring system 1 assists the training of the subject P by analyzing, evaluating and managing the measurement results. The motion state monitoring system 1 displays the analysis, evaluation and management measurement results on the display unit. The subject P wears the sensor 200 at a predetermined part of the body and performs a motion test. For example, the motion test is a motion function test that measures the motion state of the part when the subject P performs a specified motion and measures the motor function. The motion state monitoring system 1 includes, for example, a training assistance system.

[0037] In the following, the specified action is referred to as the monitoring object action. The monitoring object action is specified in correspondence with a body part. For example, the monitoring object action includes shoulder flexion and extension, shoulder adduction and abduction, shoulder internal and external rotation, neck flexion and extension, neck rotation, elbow flexion and extension, hip joint internal and external rotation, forearm pronation and supination, and thoracolumbar lateral flexion. The monitoring object action can also be specified by left and right distinction when the part is either left or right. The part of the monitoring object is referred to as the object part. As the object part, one or more object parts can be corresponding to one monitoring object action, and the same object part can be corresponding to different monitoring object actions.

[0038] like Figure 1As shown, the motion state monitoring system 1 includes a measuring instrument 2 and a motion state monitoring device 4. The motion state monitoring device 4 may also be referred to as the motion state monitoring system 1. The motion state monitoring device 4 is a device for monitoring the motion state of the target part of the body of the subject P. Hereinafter, the motion state of the target part of the body of the subject P is referred to as the motion state of the target part. The motion state monitoring system 1 may also include a photographing unit 3. In addition, the measuring instruments 2-1, measuring instruments 2-2, ..., etc. shown in the figure are collectively referred to as measuring instruments 2.

[0039] The measuring instrument 2 measures the movement state of the object part. Specifically, the measuring instrument 2 is a measuring device that measures the moving direction and movement amount of the object part. The measuring instrument 2 has a sensor 200 that detects the movement state. In the present embodiment, the measuring instrument 2 may have an acceleration sensor and an angular velocity sensor. The measuring instrument 2 measures its own acceleration and angular velocity. Specifically, the measuring instrument 2 may also include a three-axis acceleration sensor and a three-axis angular velocity sensor. In this case, the measuring instrument 2 measures the movement amount in the three-axis directions of the XYZ axis and the rotation angle around the three axes. In addition, the measuring axes are not limited to three axes, but may also be less than two axes. In addition, the measuring instrument 2 may also have a geomagnetic sensor that detects geomagnetism and measures the direction in which it is facing. In addition, the illustrated sensors 200-1, sensor 200-2, ..., etc. are collectively referred to as sensors 200.

[0040] Each measuring instrument 2 is connected to the operation status monitoring device 4 in a state capable of communicating. In the present embodiment, the communication between each measuring instrument 2 and the operation status monitoring device 4 is short-range wireless communication such as Bluetooth (registered trademark), NFC (NearField Communication) and ZigBee. However, the communication is not limited thereto, and may also be wireless communication via a network such as a wireless LAN (Local Area Network). In addition, the communication may also be wired communication via the Internet, LAN, WAN (Wide Area Network) or a network formed in a manner including a combination of these.

[0041] The measuring instrument 2 has a wearing mechanism for the sensor 200 in addition to the sensor 200. Moreover, the sensor 200 is worn at a wearing position 20 corresponding to the target part of the body of the subject P via the wearing mechanism. In addition, in order to correspond to the measurement of various monitoring object actions, a plurality of sensors 200 are respectively associated with the target parts of the body of the subject P, and can be worn on the target parts with which the association is established. In the accompanying drawings, the wearable target parts are shown by wearing positions 20-1, 20-2, ..., 20-11. Each wearing position 20-1, 20-2, ..., 20-11 is associated with each sensor 200-1, 200-2, ..., 200-11. For example, the wearing positions 20-1, 20-2, ..., 20-11 are respectively called the right upper arm, the right forearm, the head, the chest (trunk), the waist (pelvis), the left upper arm, the left forearm, the right thigh, the right calf, the left thigh, and the left calf. The association relationship between the wearing position 20 and the sensor 200 is established by pairing the sensor 200 with the motion state monitoring device 4 in advance and establishing a correspondence between the identification information (ID) of the wearing position 20 and the ID of the sensor 200 on the application of the motion state monitoring device 4. In addition, the wearing positions 20-1, 20-2, ..., etc. are collectively referred to as the wearing positions 20.

[0042] In this embodiment, the wearing position 20 used in the motion test is selected from the wearing positions 20-1 to 20-11 according to the monitored object action selected by the user. In addition, the user is a user who uses the action status monitoring device 4, such as the subject P himself or a staff member who implements the motion test. Then, the subject P or the staff member wears the sensor 200 (in this figure, sensors 200-1, 200-2, 200-6, 20-7) that has established an association relationship with the selected wearing position 20 (in this figure, wearing positions 20-1, 20-2, 20-6, 20-7) of the subject P's body, and starts the motion test.

[0043] In addition, although a plurality of sensors 200 are prepared that are respectively associated with a plurality of wearing positions 20, the number of wearing positions 20 to be prepared may be one. In addition, the number of sensors 200 to be prepared may also be one.

[0044] The sensor 200 starts measuring according to the start of the motion test and sends the measurement information to the motion state monitoring device 4. The measurement information may include acceleration information, angular velocity information or quaternion information. In addition, the measurement information may also include the components of each measurement axis direction (X-axis direction, Y-axis direction, Z-axis direction) mentioned above. Moreover, the sensor 200 stops measuring according to the end of the motion test.

[0045] The photographing unit 3 photographs the motion state of the object part. The photographing unit 3 outputs the photographed image to the motion state monitoring device 4. The photographing unit 3 includes, for example, a camera. The photographing unit 3 can shoot both video and still images. Therefore, the image can include both video and still images. The image of the motion state of the object part of the body of the subject P is called the subject captured image. Sometimes the subject captured image is simply referred to as the captured image.

[0046] The imaging unit 3 and the operating state monitoring device 4 are connected in a communicable state. In this embodiment, the communication between the imaging unit 3 and the operating state monitoring device 4 can be selected from various wireless communications and wired communications described in the communication between the measuring device 2 and the operating state monitoring device 4 .

[0047] In addition, the imaging unit 3 may not be a necessary structure of the motion state monitoring system 1. For example, if the motion state monitoring device 4 can obtain the image of the subject, the imaging unit 3 may not be provided. For example, the image of the subject may be pre-stored in a predetermined storage device (e.g., Fig.13 In the storage device STR), the action status monitoring device 4 obtains the image of the subject from the storage device.

[0048] The motion state monitoring device 4 monitors the motion state of the object part during the motion test. The motion state monitoring device 4 analyzes, evaluates and manages the information related to the motion state. The motion state monitoring device 4 includes, for example, a computer device. Specifically, the motion state monitoring device 4 may include a personal computer, a notebook computer, a mobile phone, a smart phone, a tablet terminal or other communication terminal devices capable of inputting and outputting data. In addition, the motion state monitoring device 4 may also include a server computer. In this embodiment, the motion state monitoring device 4 is set as a tablet terminal for illustration.

[0049] The motion state monitoring device 4 is used by the user during and before and after the exercise test. The motion state monitoring device 4 receives the user's selection of the monitored motion and notifies the user of the wearing position 20 corresponding to the target part. In addition, the motion state monitoring device 4 sends a measurement start or measurement stop request to the sensor 200 according to the start or end of the exercise test.

[0050] In addition, the action state monitoring device 4 outputs measurement-related information as a measurement result according to the measurement information received from the sensor 200. Here, the measurement-related information represents information related to the measurement information, and may include the measurement information itself. In addition, the measurement-related information may also be information that has undergone various transformations on the measurement information. Moreover, the measurement-related information may also be information that has undergone calculations based on the measurement information. The measurement-related information may also include information related to the above-mentioned action state. In addition, the information related to the above-mentioned action state may also include the measurement-related information. That is, the information related to the above-mentioned action state may be information based on the measurement-related information, or may include the measurement-related information itself.

[0051] The action state monitoring device 4 can also be connected to an external server (not shown) via a network in a communicative manner. The external server can be a computer device or a cloud server on the Internet. In this case, the action state monitoring device 4 can send the measurement-related information of the subject P or the information related to the action state stored by itself to the external server.

[0052] Figure 2 as well as Figure 3 FIG. 2 is a structural diagram illustrating how the sensor 200 of the measuring instrument 2 is worn in the operation state monitoring system 1 according to the first embodiment. Figure 2 As shown, the measuring device 2 includes a sensor 200, a wearing pad 210, and a belt 220. The belt 220 is formed so as to be able to be wrapped around a target part of the subject P. The sensor 200 is incorporated in the wearing pad 210, for example. The wearing pad 210 in which the sensor 200 is incorporated is formed so as to be attachable to and detachable from the belt 220.

[0053] like Figure 3 As shown, the band 220 is wrapped around the right upper arm, which is one of the target parts of the subject P. The sensor 200 is attached to the band 220 via the wearing pad 210 after pairing and calibration are completed.

[0054] Figure 4 The block diagram is a diagram illustrating a measuring instrument 2 and an action state monitoring device 4 in the action state monitoring system 1 involved in the first embodiment. As described above, the action state monitoring system 1 includes a measuring instrument 2 and an action state monitoring device 4. The measuring instrument 2 has a sensor 200. In the figure, the sensor 200 is a sensor 200 that has established an association relationship with a wearing position 20 selected based on the action of the monitored object among the prepared sensors 200-1 to 200-11. The sensor 200 is set to be paired with the action state monitoring device 4 in advance and calibrated. In addition, the number of sensors 200 is not limited to one, and it can also be two or more.

[0055] The operation state monitoring device 4 includes an acquisition unit 41, a calculation processing unit 42, an image processing unit 43, a display unit 44, a display control unit 45, and a receiving unit 46. The acquisition unit 41, the calculation processing unit 42, the image processing unit 43, the display unit 44, the display control unit 45, and the receiving unit 46 have functions as an acquisition unit, a calculation processing unit, an image processing unit, a display unit, a display control unit, and a receiving unit.

[0056] The acquisition unit 41 acquires the detection result of the sensor 200. Specifically, the acquisition unit 41 acquires the measurement information from the sensor 200 worn on the target part. In addition, the acquisition unit 41 acquires the image captured by the subject. For example, the acquisition unit 41 acquires the image captured by the subject from the imaging unit 3. In addition, the acquisition unit 41 can also acquire the image captured by the subject from a storage device storing the image captured by the subject.

[0057] In addition, the acquisition unit 41 can also acquire an image that draws the movement state of the object part. The image that draws the movement state of the object part is called a subject drawing image. In addition, the subject drawing image is sometimes simply referred to as a drawing image. The image processing unit 43 can also generate a subject drawing image based on the captured image of the subject. The acquisition unit 41 acquires the subject drawing image from the image processing unit 43. In addition, the acquisition unit 41 can also acquire the subject drawing image from a storage device that stores the subject drawing image. In addition, the acquisition unit 41 acquires information input from the user through the acceptance unit 46.

[0058] The calculation processing unit 42 generates a calculation result indicating the motion state of the target part of the body of the subject P based on the detection result of the sensor 200. The detection result of the sensor 200 also includes the angles of the joints of the body of the subject P detected in the monitored motion, the angles of the joints in an arbitrary coordinate system measured based on the detection result of an arbitrary sensor, and the like. Hereinafter, the generation of the calculation result indicating the motion state of the monitored motion is also referred to as the measurement of the monitored motion.

[0059] For example, the calculation processing unit 42 performs calculation processing based on the detection results of the sensor 200-1 worn on the right upper arm (wearing position 20-1) of the subject P and the sensor 200-2 worn on the right forearm (wearing position 20-2) among the sensors 200-1 to 200-11. As a result, the calculation processing unit 42 generates a calculation result indicating the action state of the right elbow flexion and extension action of the subject P.

[0060] Alternatively, the calculation processing unit 42 performs calculation processing based on the detection results of the sensor 200-5 worn on the waist (wearing position 20-5) of the subject P and the sensor 200-8 worn on the right thigh (wearing position 20-8) among the sensors 200-1 to 200-11. As a result, the calculation processing unit 42 generates a calculation result indicating the movement state of the right side waist lateral flexion movement of the subject P.

[0061] Alternatively, the calculation processing unit 42 may perform calculation processing using a learned model generated by machine learning using past detection results of the sensor 200. The calculation processing unit 42 can perform calculation processing using such a learned model to more accurately calculate whether the motion state of the monitored motion of the subject P is good or not.

[0062] The calculation processing unit 42 may also perform processing to visualize the calculation result in a graph or the like. This allows the user to easily understand the motion state of the target part. Therefore, for example, it can help in the care of the subject P.

[0063] The image processing unit 43 processes the captured image of the subject. In addition, the image processing unit 43 processes the rendered image of the subject. The image processing unit 43 performs image processing on at least one of the captured image of the subject and the rendered image of the subject.

[0064] The image processing unit 43 may also generate a subject rendered image of a 3D model that draws the motion state of the subject P. The image processing unit 43 may also generate a subject rendered image of a 3D model that draws the motion state captured in the image of the subject. In addition, the image processing unit 43 may also pre-generate a subject rendered image of a 3D model that draws a motion state that serves as a reference. The motion state that serves as a reference includes an ideal motion state of the object part of the subject P. In addition, the image processing unit 43 may also generate a subject rendered image of an avatar instead of a 3D model. The avatar may also include an animated image. In this way, the subject rendered image is not necessarily limited to a 3D model, but may also include a rendered image of an avatar.

[0065] The image processing unit 43 may also generate a trajectory of the object part in the drawn image of the subject of the 3D model. For example, the image processing unit 43 may also generate a trajectory of the object part in the drawn image of the subject of the 3D model in which the action state captured in the captured image of the subject is drawn. In addition, the image processing unit 43 may also generate a trajectory of the object part in the captured image of the subject. In addition, the image processing unit 43 may also generate a trajectory of the object part in the drawn image of the subject of the 3D model in which a pre-generated action state serving as a reference is drawn.

[0066] The image processing unit 43 may also generate a Lissajous figure synthesizing a plurality of trajectories. Specifically, for example, as a first trajectory, the image processing unit 43 generates a trajectory of an object part in a drawn image of the subject, in which a 3D model of an action state captured in a photographed image of the subject is drawn. In addition, as a second trajectory, the image processing unit 43 generates a trajectory of an object part in a drawn image of the subject, in which a 3D model of an action state that is generated in advance and serves as a reference is drawn. Furthermore, the image processing unit 43 generates a Lissajous figure synthesizing the first trajectory and the second trajectory.

[0067] The image processing unit 43 may generate a graphic representing a movable region of a target part in the subject rendering image of the 3D model.

[0068] The display unit 44 displays the calculation result of the motion state of the object part generated by the calculation processing unit 42. In addition, the display unit 44 displays at least one of the captured image of the subject and the rendered image of the subject that has been image-processed by the image processing unit 43. For example, the display unit 44 displays the rendered image of the subject of the 3D model generated by the image processing unit 43. The display unit 44 can enlarge or reduce the rendered image of the subject of the 3D model for display, and can also enlarge or reduce the captured image of the subject for display.

[0069] The display unit 44 displays the rendered image of the subject including the trajectory of the target part generated by the image processing unit 43. The display unit 44 may also display the captured image of the subject including the trajectory of the target part generated by the image processing unit 43. The display unit 44 displays at least one of the captured image of the subject and the rendered image of the subject in synchronization with the calculation result.

[0070] The display unit 44 may display the Lissajous figure generated by the image processing unit 43. In addition, the display unit 44 may display a figure indicating the generated movable region. The display unit 44 may display the Lissajous figure and the figure indicating the movable region synchronously with the calculation result.

[0071] The display unit 44 may also display at least one of the vital sign data and the walking speed. In addition, the display unit 44 may also display a plurality of photographed images of the subject and the drawn images of the subject at different times. The display unit 44 displays the calculation results and the photographed images of the subject and the drawn images of the subject under the control of the display control unit 45.

[0072] The display control unit 45 controls the display of the display unit 44. Specifically, the display control unit 45 controls the display unit 44 to display the calculation results generated by the calculation processing unit 42, and the subject's photographed images and subject's rendered images processed by the image processing unit 43.

[0073] The receiving unit 46 receives information input by the user.

[0074] Next, an operation state monitoring method using the operation state monitoring system 1 will be described. Figure 5 This is a flowchart illustrating a measurement method in the operation state monitoring method using the operation state monitoring system 1 according to the first embodiment. Figure 6 This is a flowchart illustrating a display method in the operation state monitoring method using the operation state monitoring system 1 according to the first embodiment.

[0075] like Figure 5 As shown in step S11 of FIG. 1 , in the operation state monitoring system 1 , a correspondence relationship is established between the measuring device 2 and the wearing position 20 . Thus, a pairing process can be performed between the operation state monitoring device 4 and the measuring device 2 .

[0076] Next, as shown in step S12, the calibration of the sensor 200 in the measuring instrument 2 is performed. Calibration refers to, for example, measuring the output value (error component) of the sensor 200 in a stationary state for measuring the movement of the monitoring object, and subtracting the error component from the measured value. Here, the output value of the sensor 200 is set to be stable after a predetermined period (about 20 seconds) from when the sensor 200 is stationary. In this case, it is preferred that, in the calibration, the output value of the sensor 200 after a predetermined period from when the sensor 200 is stationary is used as the error component. Therefore, in the present embodiment, the output value of the sensor 200 after a predetermined period from when the sensor 200 is stationary and the user indicates to start calibration is used as the error component. In addition, calibration refers to the processing period until the error component is determined, and calibration completion refers to the situation where the output value (error component) of the sensor in the stationary state has been determined.

[0077] During calibration, for example, "calibration in progress" is displayed on the display unit 44. When calibration is completed, "calibration completed" is displayed on the display unit 44. In addition, the notification of calibration in progress and calibration completed is not limited to the case of being displayed on the display unit 44, and other notification methods such as voice notification may be used for notification.

[0078] Next, as shown in step S13, the measuring device 2 is worn on the subject P. In the present embodiment, the sensor 200-1, sensor 200-2, sensor 200-6, and sensor 200-7 of the sensors 200-1 to 200-11 are respectively worn on the right upper arm (wearing position 20-1), right forearm (wearing position 20-2), left upper arm (wearing position 20-6), and left forearm (wearing position 20-7) of the subject P. The wearing of the sensor 200 may also be implemented via the wearing pad 210 and the belt 220.

[0079] Next, as shown in step S14 , the monitoring target action that can be measured using the sensor 200 worn on the subject P is measured among the plurality of monitoring target actions.

[0080] Next, as shown in step S15 , the operation state monitoring device 4 starts measuring the operation state of the target part.

[0081] Next, a display method will be described. The display by the display unit 44 of the present embodiment may be performed during the measurement in the exercise test, or may be performed before the measurement in the exercise test starts or after the measurement ends.

[0082] like Figure 6 As shown in step S21, based on the detection result of the sensor that detects the motion state of the target part, the calculation processing unit 42 generates a calculation result indicating the motion state. The calculation result indicating the generated motion state includes, for example, a measurement result of the motion of the monitoring target.

[0083] Next, as shown in step S22 , the image processing unit 43 performs image processing on at least one of the captured image of the subject in which the motion state of the target part is captured and the rendered image of the subject in which the motion state is rendered.

[0084] Next, as shown in step S23 , the display unit 44 displays the image processed in synchronization with the calculation result.

[0085] Figure 7 to Figure 12 FIG. 4 is a diagram illustrating the display of the display unit 44 in the operation state monitoring system 1 according to the first embodiment. Figure 7 , a display image 300-1 before the start of measurement in the exercise test is shown. Figure 7 As shown, the display image 300 - 1 includes display areas 302 , 304 , 305 , 306 , 309 , and 310 .

[0086] Icon images representing a plurality of wearing positions 20 that are candidates for wearing the sensor 200 are displayed in the display area 302. The wearing position 20 (positions shown as "1", "2", "6", and "7" in the figure) corresponding to the selected measurement action may also be highlighted in the display area 302. Thus, the user can easily visually confirm the wearing position 20, and thus can smoothly perform the exercise test.

[0087] The rotation angles of the sensors 200-1, 200-2, ..., 200-11 associated with the wearing positions 20-1, 20-2, ..., 20-11 are two-dimensionally displayed in the display area 304. The rotation angles displayed here dynamically change according to the movement of the sensor 200 linked to the movement of the subject P. Therefore, the user can identify the sensor 200 whose power is turned off or the sensor 200 that is not operating normally through the display area 304 before starting the measurement.

[0088] In the case where a plurality of sensors 200 are used in the exercise test, input operation buttons for collectively calibrating the plurality of sensors 200 are displayed in the display area 305 . This allows the user to easily request calibration of each of the plurality of sensors 200 via the display area 305 .

[0089] An input operation button for starting an exercise test, that is, starting measurement by the sensor 200 , is displayed in the display area 306 . This allows the user to easily request the start of measurement by the sensor 200 via the display area 306 .

[0090] The display area 309 displays the measurement-related information of each sensor 200 used. Before the measurement starts, the measurement-related information is not displayed. The display area 310 displays the motion state index of the target part of each monitored target action being performed. Before the measurement starts, the motion state index of the target part is not displayed.

[0091] exist Figure 8 , a display image 300-2 is shown when the measurement in the exercise test is completed. The display image 300-2 includes display areas 302, 304, 305, 306, 309, and 310, similarly to the display image 300-1. The display areas 302 and 304 of the display image 300-2 are Figure 7 The display areas 302 and 304 of the displayed image 300 - 1 are shown to be the same.

[0092] The display area 308 displays an input operation button for ending the exercise test, that is, stopping the measurement by the sensor 200 . This allows the user to easily request the stop of the measurement by the sensor 200 via the display area 308 .

[0093] The display area 309 displays measurement-related information of each sensor 200 used. The surrounding X based on the output of a part of the sensors 200-1, 200-2, 200-6, and 200-7 used, is displayed in time series. S , Y S and Z S The rotation angle of the axis.

[0094] The display area 310 displays the motion state index of the target part for each implemented monitored object action. The motion state index is an index that indicates the motion state of the target part when the monitored object action is implemented. The operation processing unit 42 calculates the motion state index of the target part based on the measurement-related information of the sensor 200. For example, when the monitored object action is "right elbow flexion and extension", the measurement-related information of the sensors 200-1 and 200-2 at the wearing positions 20-1 and 20-2 is used. In this case, the operation processing unit 42 can calculate the motion state index based on the difference in the measurement-related information of the sensors 200-1 and 200-2. The display area 310 displays the motion state index of a part of the monitored object actions in the implemented monitored object actions in a time series.

[0095] exist Fig. 9 The display image 300-3 is shown in FIG. The display image 300-3 includes display areas 311 to 314. The image captured by the subject is displayed on the display area 311. The image captured by the subject may also include a video. The rendered image of the subject is displayed on the display area 312. The rendered image of the subject displayed on the display area 312 includes a rendered image of the subject of a 3D model generated by the image processing unit 43. The rendered image of the subject of a 3D model can be generated by drawing the action state captured in the image captured by the subject. On the display area 313, based on the detection result of the sensor 200, the calculation result representing the action state generated by the calculation processing unit 42 is displayed. The calculation result includes, for example, displaying the time change of the rotation angle in a graphical manner.

[0096] In this way, the display unit 44 can also display the captured image of the subject and the drawn image of the subject. As the drawn image of the subject, the display unit 44 can also display the drawn image of the 3D model. The display unit 44 can also display the captured image of the subject and the drawn image of the subject synchronously. In other words, the shooting time of the captured image of the subject and the shooting time of the captured image of the subject used when generating the drawn image of the subject can also be displayed synchronously. Thus, the user can correspond to the action state of the observed object part through the 3D model. Therefore, the action state can be easily understood.

[0097] The display unit 44 can display the captured image of the subject in the display area 311 by enlarging or reducing it, and can also display the rendered image of the subject in the display area 312 by enlarging or reducing it. The display unit 44 can also display the rendered image of the subject in a 3D model by enlarging or reducing it. This makes it easier to understand the action status.

[0098] The display area 312 displays a graphic 312a representing the movable area of ​​the object part in the subject rendering image of the 3D model generated by the image processing unit 43. In addition, the vital sign data and the walking speed are displayed in the display area 314. In this way, the display unit 44 can display the generated graphic 312a representing the movable area, and can also further display at least one of the vital sign data and the walking speed. By such a display method, the action state can be grasped in more detail.

[0099] exist Fig.10 3 shows a display image 300-4. The display image 300-4 includes display areas 311, 313, and 315. In the display area 315, a calculation result representing the operation state generated by the calculation processing unit 42 is displayed similarly to the display area 313. In the display area 315, a plurality of calculation results are combined and displayed in an enlarged manner.

[0100] In this way, the display unit 44 can also display the photographed image of the subject and the calculation result. In addition, the display unit 44 can also display the drawn image of the subject and the calculation result instead of the photographed image of the subject, or on the basis of the photographed image of the subject. The display unit 44 can also display at least one of the photographed image of the subject and the drawn image of the subject synchronously with the calculation result. In other words, the shooting time of the photographed image of the subject and the measurement time performed by the sensor 200 can also be displayed synchronously. In this way, the user can correspond to the action state of the observed object part through the 3D model, so that the action state can be easily understood.

[0101] exist Fig.11 Display image 300-5 is shown in . Display image 300-5 includes display areas 316 to 319. On display area 316, an image of the subject captured at a predetermined capturing time is displayed. For example, an image of the subject captured including a motion state in a case where the motion state of the object part is not good is displayed. On display area 317, the calculation result is displayed synchronously with the image of the subject captured in display area 316. On display area 318, an image of the subject captured at a capturing time different from the predetermined capturing time is displayed. For example, an image of the subject captured after the motion state is well restored is displayed. On display area 319, the calculation result is displayed synchronously with the image of the subject captured in display area 318.

[0102] Thus, the display unit 44 can also display a plurality of subject images captured at different times. In addition, the display unit 44 can also display a subject drawn image on which a plurality of subject images captured at different times are drawn, or can also display subject captured images and subject drawn images at different times.

[0103] exist Fig.12Display image 300-6 is shown in FIG. Display image 300-6 includes display areas 320 to 324. A 3D model of the subject rendered image generated by the image processing unit 43 is displayed on display area 320. The subject rendered image in display area 320 includes a trajectory 320a of the object part. The subject rendered image displayed on display area 320 may also be a 3D model of the subject rendered in the action state captured in the captured image. On display area 321, the calculation result is displayed synchronously with the subject rendered image in display area 320.

[0104] Another 3D model of the subject rendered image is displayed on the display area 322. The subject rendered image on the display area 322 includes a trajectory 322a of the object part. The subject rendered image displayed on the display area 322 may also be a subject rendered image in which a 3D model of a motion state that is pre-generated and serves as a reference is rendered by the image processing unit 43. On the display area 323, the calculation result is displayed synchronously with the subject rendered image on the display area 320.

[0105] In this way, the display unit 44 can also display the subject rendering image including the trajectory 320a and 322a of the object part in the subject rendering image of the 3D model. In this case, as the subject rendering image including the trajectory 320a and 322a, the display unit 44 can display both the subject rendering image in which the action state captured in the captured image is drawn, and the subject rendering image in which the 3D model of the action state generated in advance as a reference is drawn. In addition, the display unit 44 can also display the subject captured image including the trajectory generated by the image processing unit 43 and being the trajectory of the object part in the captured image.

[0106] The Lissajous figure generated by the image processing unit 43 is displayed in the display area 324. The Lissajous figure is generated by synthesizing the first trajectory and the second trajectory. The first trajectory is, for example, a trajectory 320a of the target part as displayed in the display area 320. The second trajectory is, for example, a trajectory 322a of the target part as displayed in the display area 322.

[0107] Thus, the display unit 44 may display the Lissajous figure synthesized by the first trajectory and the second trajectory generated by the image processing unit 43. By displaying the Lissajous figure, the difference between the pre-generated reference motion state and the motion state at the time of shooting can be displayed, and the motion state can be easily understood.

[0108] Next, the effects of this embodiment will be described. The motion state monitoring system 1 of this embodiment displays at least one of the photographed image of the subject and the rendered image of the subject in synchronization with the calculation result. Therefore, the motion state of the target part can be easily understood.

[0109] The display unit 44 displays the rendered image of the subject of the 3D model, so that the motion state can be easily observed. For example, the display unit 44 can display the rendered image of the subject in an enlarged or reduced manner. Therefore, it is possible to focus on each object part and to easily understand the motion state. In addition, the display unit 44 can grasp the motion state in more detail by displaying a graph representing the movable area of ​​the object part, vital sign data, walking speed, etc.

[0110] The display unit 44 displays the trajectory and Lissajous figure of the target part in the operating state. This enables the operating state of the target part to be understood in more detail.

[0111] In the above-mentioned embodiment, the present disclosure is described as a hardware configuration, but the present disclosure is not limited thereto. The present disclosure can also implement each process involved in the operation state monitoring method by causing a processor to execute a computer program, for example, an operation state monitoring program.

[0112] In the above-mentioned embodiment, the computer is composed of a computer system including a personal computer, a word processor, etc. However, the present invention is not limited thereto, and the computer can also be composed of a server of a LAN, a host computer (personal computer) communication, a computer system connected to the Internet, etc. In addition, it is also possible to disperse the functions of each device on the network and constitute the computer by the entire network.

[0113] Fig.13 The schematic structure diagram of the operation state monitoring device 4 including a computer according to the embodiment is shown as an example. Fig.13 As shown, the action state monitoring device 4 may also include a processor PRC, a memory MMR, a storage device STR, and a user interface UI. The storage device STR stores the processing performed by each structure of the action state monitoring device 4 as a program. In addition, the processor PRC reads the program from the storage device STR into the memory MMR and executes the program. Thus, the processor PRC implements the functions of each structure in the action state monitoring device 4. The user interface UI may also include input devices such as a keyboard, a mouse, and a camera, and output devices such as a display, a printer, and a speaker.

[0114] Each structure of the action state monitoring device 4 can also be implemented by dedicated hardware. In addition, part or all of each structural element can also be implemented by a general or dedicated circuit (Circuitry), a processor PRC, etc. or a combination thereof. These can be composed of a single chip or a plurality of chips connected via a bus. Part or all of each structural element can also be implemented by a combination of the above-mentioned circuits, etc. and programs. In addition, as a processor PRC, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-programmable Gate Array), a quantum processor (quantum computer control chip), etc. can be used.

[0115] In addition, when part or all of the structural elements of the action state monitoring device 4 are implemented by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. can be configured in a centralized manner or in a decentralized manner. For example, the information processing devices, circuits, etc. can also be implemented by a cloud server system, a cloud computing system, etc. as each connected via a communication network NW. In addition, the function of the action state monitoring device 4 can also be provided in the form of SaaS (Software as a Service).

[0116] The execution order of each process in the apparatus and method shown in the technical solution, the specification and the drawings can be implemented in any order unless it is specifically stated as "earlier than", "prior to", etc., and the output of the previous process is not used in the subsequent process. Even if the action flow in the technical solution, the specification and the drawings is described using "first", "next", etc. for convenience, it does not mean that it must be implemented in this order.

[0117] Any type of non-temporary computer-readable medium can be used to store the program and provide it to the computer. Non-temporary computer-readable media include any type of tangible storage media. Examples of non-temporary computer-readable media include: magnetic storage media (such as floppy disks, magnetic tapes, hard disk drives, etc.), optical magnetic storage media (such as optical magnetic disks), CD-ROM (Compact Disc Read Only Memory), CD-R (Recordable Compact Disc), CD-R / W (Rewritable Compact Disc), semiconductor memory (such as mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), Flash ROM, RAM (Random Access Memory), etc.). Any type of temporary computer-readable medium can be used to provide the program to the computer. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can provide the program to the computer via a wired communication line (such as wires and optical fibers) or a wireless communication line.

[0118] From the present disclosure described in this way, it is obvious that the embodiments of the present disclosure can be changed in many ways. Such changes should not be considered as departing from the spirit and scope of the present disclosure, and it is obvious to those skilled in the art that all such modifications are included in the scope of the technical solution.

Claims

1. An action state monitoring system for monitoring an action state of a target part of a body of a subject, comprising: an operation processing unit that generates an operation result representing the operation state based on a detection result of a sensor that detects the operation state; an image processing unit that performs image processing on at least one of a captured image capturing the action state and a drawn image drawing the action state; A display unit displays the image subjected to image processing in synchronization with the calculation result.

2. The operating status monitoring system according to claim 1, wherein: The image processing unit generates the rendered image in which the 3D model in the action state is rendered or the rendered image in which the avatar in the action state is rendered, The display unit displays the generated rendering image of the 3D model or the avatar.

3. The operating state monitoring system according to claim 2, wherein: The image processing unit generates a trajectory of the object part in the rendered image of the 3D model or the avatar, The display unit displays the drawn image including the generated trajectory.

4. The operating state monitoring system according to claim 2, wherein: The image processing unit generates a trajectory of the target part in the captured image. The display unit displays the captured image including the generated trajectory.

5. The operating state monitoring system according to claim 2, wherein: The image processing unit generates a trajectory of the target part in the drawn image of the 3D model or the avatar in which the action state captured in the captured image is drawn, The display unit displays the drawn image including the generated trajectory.

6. The operating status monitoring system according to claim 2, wherein: The image processing unit generates a trajectory of the target part in the rendered image in which the 3D model or the avatar of the action state generated in advance as a reference is rendered, The display unit displays the drawn image including the generated trajectory.

7. The operating state monitoring system according to claim 2, wherein: The image processing unit generates a first trajectory of the object part in the drawn image of the 3D model or the avatar in the action state captured in the captured image, and generating a second trajectory of the target part in the drawn image of the 3D model or the avatar in the action state generated in advance as a reference, and generating a Lissajous figure synthesizing the first trajectory and the second trajectory, The display unit displays the generated Lissajous figure.

8. The operating status monitoring system according to claim 2, wherein: The display unit displays the drawn image of the 3D model or the avatar by enlarging or reducing it.

9. The operating state monitoring system according to claim 2, wherein: The image processing unit generates a graphic representing a movable area of ​​the target part in the rendered image of the 3D model or the avatar, The display unit displays the generated graphic representing the movable area.

10. The operating status monitoring system according to claim 1, wherein: The display unit further displays at least any one of vital sign data and walking speed.

11. The operating status monitoring system according to claim 1, wherein: The display unit displays the plurality of images at different times.

12. A motion state monitoring method, which uses a motion state monitoring system to monitor the motion state of a target part of a subject's body, wherein: The operation status monitoring system comprises: an operation processing unit that generates an operation result representing the operation state based on a detection result of a sensor that detects the operation state; an image processing unit that performs image processing on at least one of a captured image capturing the action state and a drawn image drawing the action state; a display unit for synchronously displaying the image subjected to image processing and the calculation result, The action state monitoring method comprises: A step of generating the calculation result representing the action state based on the detection result; a step of performing image processing of the image; The step of displaying the image that has been subjected to image processing synchronously with the calculation result.

13. An action state monitoring program product, which causes a computer included in an action state monitoring system to execute a process for monitoring the action state of a target part of a subject's body, wherein: The operation status monitoring system comprises: an operation processing unit that generates an operation result representing the operation state based on a detection result of a sensor that detects the operation state; an image processing unit that performs image processing on at least one of a captured image capturing the action state and a drawn image drawing the action state; a display unit for synchronously displaying the image subjected to image processing and the calculation result, The operation status monitoring program product enables the computer to execute the following steps, namely: A step of generating the calculation result representing the action state based on the detection result; a step of performing image processing of the image; The step of displaying the image that has been subjected to image processing synchronously with the calculation result.

Images