Operating state monitoring system, operating state monitoring method and operating state monitoring program

Abstract

To provide an operating state monitoring system, an operating state monitoring method and an operating state monitoring program capable of making an operating state in an object part of a subject's body easy to understand.SOLUTION: An operating state monitoring system is an operating state monitoring system 1 for monitoring an operating state in an object part of the body of a subject P and comprises: an arithmetic processing part 42 for generating an arithmetic result on the basis of a detection result by a sensor 200 for detecting the operating state; an image processing part 43 for performing image processing on at least one of a taken image of the operating state and a drawn image of the operating state; and a display part 44 for synchronously displaying the processed image and the arithmetic result.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to an operation status monitoring system, an operation status monitoring method, and an operation status monitoring program. [Background technology]

[0002] Patent Document 1 discloses a motion status monitoring system that monitors the motion status of a target part of a subject's body. The display unit in the motion status monitoring system of Patent Document 1 displays an icon indicating the position of a sensor that detects the motion status, and a line graph showing the motion status detected by the sensor as an angle from a predetermined position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-034450 Summary of the Invention [Problem to be solved by the invention]

[0004] When the data is reviewed, it is difficult to understand the motion state of the target part of the subject's body by simply displaying the angle using a line graph. It is desired to make it easier to understand the motion state of the target part of the subject's body.

[0005] The present disclosure has been made in consideration of the above background, and aims to provide a motion status monitoring system, a motion status monitoring method, and a motion status monitoring program that make it easier to understand the motion status of a target part of a subject's body when reviewing the data. [Means for solving the problem]

[0006] A motion state monitoring system according to an embodiment of the present disclosure is a motion state monitoring system for monitoring a motion state of a target part of a subject's body, and includes: a calculation processing unit that generates a calculation result representing the motion state based on a detection result by a sensor that detects the motion state, an image processing unit that performs image processing of at least one of an image captured of the motion state and an image drawn of the motion state, and a display unit that displays the image processed by the image processing in synchronization with the calculation result. The calculation processing unit may perform the calculation processing using a trained model generated by machine learning using past detection results of the sensor.

[0007] In the above operating state monitoring system, the image processing unit may generate the drawn image of a 3D model depicting the operating state or the drawn image of an avatar depicting the operating state, and the display unit may display the drawn image of the generated 3D model.

[0008] In the above operating status monitoring system, the image processing unit may generate a trajectory of the target part in the drawn image of the 3D model or the avatar, and the display unit may display the drawn image including the generated trajectory.

[0009] In the above operational state monitoring system, the image processing unit may generate a trajectory of the target part in the captured image, and the display unit may display the captured image including the generated trajectory.

[0010] In the above-mentioned operation status monitoring system, the image processing unit may generate a trajectory of the target part in the drawn image of the 3D model or the avatar that depicts the operation status captured in the captured image, and the display unit may display the drawn image including the generated trajectory.

[0011] In the above-mentioned operation status monitoring system, the image processing unit may generate a trajectory of the target part in the drawn image of the 3D model or the avatar that depicts the reference operation state that has been generated in advance, and the display unit may display the drawn image including the generated trajectory.

[0012] In the above-mentioned operation status monitoring system, the image processing unit may generate a first trajectory of the target part in the drawn image of the 3D model or the avatar that depicts the operation status captured in the captured image, generate a second trajectory of the target part in the drawn image of the 3D model or the avatar that depicts the operation status that is a reference generated in advance, generate a Lissajous figure by combining the first trajectory and the second trajectory, and the display unit may display the generated Lissajous figure.

[0013] In the above operating state monitoring system, the display unit may display the rendered image of the 3D model or the avatar by enlarging or reducing it.

[0014] In the above-mentioned motion status monitoring system, the image processing unit may generate a figure indicating a movable area of ​​the target part in the drawn image of the 3D model or the avatar, and the display unit may display the generated figure indicating the movable area.

[0015] In the above motion state monitoring system, the display unit may further display at least one of vital data and walking speed.

[0016] In the above operating status monitoring system, the display unit may display a plurality of the images taken at different times.

[0017] A motion state monitoring method according to one aspect of the present disclosure is a motion state monitoring method for monitoring a motion state of a target part of a subject's body using a motion state monitoring system, the motion state monitoring system having a calculation processing unit that generates a calculation result representing the motion state based on a detection result by a sensor that detects the motion state, an image processing unit that performs image processing of at least one of an image captured of the motion state and an image drawn of the motion state, and a display unit that displays the image-processed image and the calculation result in synchronization with each other, and includes the steps of generating the calculation result representing the motion state based on the detection result, image processing of the image, and displaying the image-processed image and the calculation result in synchronization with each other.

[0018] A motion state monitoring program according to one aspect of the present disclosure is a motion state monitoring program that causes a computer included in a motion state monitoring system to monitor the motion state of a target part of a subject's body, the motion state monitoring system having an arithmetic processing unit that generates a calculation result representing the motion state based on a detection result by a sensor that detects the motion state, an image processing unit that performs image processing of at least one of an image captured of the motion state and an image drawn of the motion state, and a display unit that displays the image-processed image and the calculation result in synchronization with each other, and causes the computer to execute the steps of generating the calculation result representing the motion state based on the detection result, image processing of the image, and displaying the image-processed image and the calculation result in synchronization with each other. Effect of the Invention

[0019] According to the present disclosure, it is possible to provide a motion status monitoring system, a motion status monitoring method, and a motion status monitoring program that make it easier to understand the motion status of a target part of a subject's body when reviewing the data. [Brief description of the drawings]

[0020] [Figure 1]1 is a configuration diagram illustrating an operation status monitoring system according to a first embodiment. [Diagram 2] 2 is a configuration diagram illustrating an example of mounting of sensors of measuring instruments in the operation status monitoring system according to the first embodiment. FIG. [Diagram 3] 2 is a configuration diagram illustrating an example of mounting of sensors of measuring instruments in the operation status monitoring system according to the first embodiment. FIG. [Figure 4] 1 is a block diagram illustrating a measuring instrument and an operation status monitoring device in an operation status monitoring system according to a first embodiment. [Diagram 5] 4 is a flow chart illustrating a measurement method in the operation state monitoring method using the operation state monitoring system according to the first embodiment. FIG. [Figure 6] 4 is a flow chart illustrating an example of a display method in the operation state monitoring method using the operation state monitoring system according to the first embodiment. FIG. [Figure 7] 4 is a diagram illustrating an example of a display on a display unit in the operational status monitoring system according to the first embodiment. FIG. [Figure 8] 4 is a diagram illustrating an example of a display on a display unit in the operational status monitoring system according to the first embodiment. FIG. [Figure 9] 4 is a diagram illustrating an example of a display on a display unit in the operational status monitoring system according to the first embodiment. FIG. [Figure 10] 4 is a diagram illustrating an example of a display on a display unit in the operational status monitoring system according to the first embodiment. FIG. [Figure 11] 4 is a diagram illustrating an example of a display on a display unit in the operational status monitoring system according to the first embodiment. FIG. [Figure 12] 4 is a diagram illustrating an example of a display on a display unit in the operational status monitoring system according to the first embodiment. FIG. [Figure 13] 1 is a schematic configuration diagram illustrating an operation state monitoring device including a computer according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, the present disclosure will be described through embodiments, but the present disclosure is not limited to the following embodiments. In addition, all of the configurations described in the embodiments are not necessarily essential as means for solving the problems. For clarity of explanation, the following descriptions and drawings are omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals, and duplicate explanations are omitted as necessary.

[0022] <Embodiment 1> A motion state monitoring system 1 according to the first embodiment will be described. FIG. 1 is a configuration diagram illustrating the motion state monitoring system according to the first embodiment. The motion state monitoring system 1 monitors the motion state of the body parts of the subject P. The motion state monitoring system 1 also supports the subject P to make the motion closer to a desired motion based on the monitoring result of the motion of the subject P. Specifically, the motion state monitoring system 1 measures the motor function of the subject P, including rehabilitation trainees and elderly people. The motion state monitoring system 1 supports the training of the subject P by analyzing, evaluating and managing the measurement results. The motion state monitoring system 1 displays the measurement results that are analyzed, evaluated and managed on a display unit. The subject P attaches the sensor 200 to a predetermined part of the body and performs an exercise test. For example, the exercise test is a motor function test that measures the motion state of the body parts when the subject P performs a specified motion and measures the motor function. The motion state monitoring system 1 includes, for example, a training support system.

[0023] In the following, the designated motion is called a monitored motion. The monitored motion is determined corresponding to a body part. For example, the monitored motion includes shoulder flexion and extension, shoulder abduction and abduction, shoulder internal and external rotation, neck flexion and extension, neck rotation, elbow flexion and extension, hip internal and external rotation, forearm pronation and supination, and thoracic and lumbar lateral bending. When the part is either the left or right, the monitored motion may be determined separately for the left and right. The part to be monitored is called a target part. As a target part, one or more target parts may be associated with one monitored motion, and the same target part may be associated with different monitored motions.

[0024] As shown in FIG. 1, the motion status monitoring system 1 includes a measuring instrument 2 and a motion status monitoring device 4. The motion status monitoring device 4 may be referred to as the motion status monitoring system 1. The motion status monitoring device 4 is a device that monitors the motion status of a target part of the body of the subject P. The motion status of the target part of the body of the subject P will hereinafter be referred to as the motion status of the target part. The motion status monitoring system 1 may include an imaging unit 3. Note that the measuring instruments 2-1, 2-2, ..., etc. shown in the figure will be collectively referred to as measuring instruments 2.

[0025] The measuring instrument 2 measures the motion state of the target part. Specifically, the measuring instrument 2 is a measuring device that measures the moving direction and amount of movement of the target part. The measuring instrument 2 has a sensor 200 that detects the motion state. In this embodiment, the measuring instrument 2 may have an acceleration sensor and an angular velocity sensor. The measuring instrument 2 measures its own acceleration and angular velocity. Specifically, the measuring instrument 2 may include a three-axis acceleration sensor and a three-axis angular velocity sensor. In this case, the measuring instrument 2 measures the amount of movement in three axial directions of the XYZ axes and the rotation angle around the three axes. Note that the measurement axes are not limited to three axes, and may be two or less axes. In addition, the measuring instrument 2 may have a geomagnetic sensor that detects geomagnetism and measures the direction in which the measuring instrument 2 is facing. Note that the sensors 200-1, 200-2, ..., etc. shown in the figure are collectively called sensors 200.

[0026] Each measuring instrument 2 is connected to the operation status monitoring device 4 in a communicable state. In this embodiment, communication between each measuring instrument 2 and the operation status monitoring device 4 is short-range wireless communication such as Bluetooth (registered trademark), NFC (Near Field Communication), and ZigBee. However, the communication is not limited to this, and may be wireless communication via a network such as a wireless LAN (Local Area Network). Furthermore, the communication may be wired communication via the Internet, a LAN, a WAN (Wide Area Network), or a network configured including a combination of these.

[0027] In addition to the sensor 200, the measuring instrument 2 has an attachment mechanism for the sensor 200. The sensor 200 is attached to an attachment position 20 corresponding to a target part of the body of the subject P via the attachment mechanism. In order to measure various monitoring target actions, each of the multiple sensors 200 is linked to a target part of the body of the subject P and can be attached to the linked target part. In the figure, the target parts to which the sensors can be attached are indicated by attachment positions 20-1, 20-2, ..., 20-11. Each attachment position 20-1, 20-2, ..., 20-11 is linked to each sensor 200-1, 200-2, ..., 200-11. For example, the attachment positions 20-1, 20-2, ..., 20-11 are respectively called the right upper arm, right forearm, head, chest (trunk), waist (pelvis), left upper arm, left forearm, right thigh, right lower leg, left thigh, and left lower leg. The attachment positions 20 and the sensor 200 are linked by previously pairing the sensor 200 with the operation state monitoring device 4 and by associating the identification information (ID) of the attachment position 20 with the ID of the sensor 200 on the application of the operation state monitoring device 4. The attachment positions 20-1, 20-2, ..., etc. are collectively called the attachment positions 20.

[0028] In this embodiment, the attachment positions 20 used in the exercise test are selected from attachment positions 20-1 to 20-11 according to the motion to be monitored selected by the user. The user is a user who uses the motion status monitoring device 4, and is, for example, the subject P himself or a staff member who performs the exercise test. Then, the subject P or the staff member attaches the sensor 200 (sensors 200-1, 200-2, 200-6, 200-7 in this figure) associated with the selected attachment position 20 (attachment positions 20-1, 20-2, 20-6, 20-7 in this figure) to the body of the subject P, and starts the exercise test.

[0029] Although a plurality of sensors 200 each associated with a respective one of a plurality of mounting positions 20 are prepared in the above embodiment, the number of mounting positions 20 prepared may be one. Also, the number of sensors 200 prepared may be one.

[0030] The sensor 200 starts measurement in response to the start of the exercise test, and transmits sensing information to the motion status monitoring device 4. The sensing information may include acceleration information, angular velocity information, or quaternion information. The sensing information may also include components in each of these measurement axis directions (X-axis direction, Y-axis direction, and Z-axis direction). Then, the sensor 200 stops measurement in response to the end of the exercise test.

[0031] The imaging unit 3 captures an image of the motion state of the target part. The imaging unit 3 outputs the captured image to the motion state monitoring device 4. The imaging unit 3 includes, for example, a camera. The imaging unit 3 may capture moving images or still images. Thus, the images may include moving images or still images. An image capturing the motion state of the target part of the body of the subject P is called a subject captured image. The subject captured image may be simply called a captured image.

[0032] The imaging unit 3 is connected in a communicable state to the operation status monitoring device 4. In this embodiment, the communication between the imaging unit 3 and the operation status monitoring device 4 may be selected from various wireless and wired communications described in the communication between the measuring instrument 2 and the operation status monitoring device 4.

[0033] The imaging unit 3 does not have to be an essential component of the operation state monitoring system 1. For example, as long as the operation state monitoring device 4 can acquire the subject captured images, the imaging unit 3 does not have to be provided. For example, the subject captured images may be stored in a predetermined storage device (for example, the storage device STR in FIG. 13), and the operation state monitoring device 4 may acquire the subject captured images from the storage device.

[0034] The motion state monitoring device 4 monitors the motion state of the target part during the exercise test. The motion state monitoring device 4 analyzes, evaluates, and manages information related to the motion state. The motion state monitoring device 4 includes, for example, a computer device. Specifically, the motion state monitoring device 4 may include a personal computer, a notebook computer, a mobile phone, a smartphone, a tablet terminal, or other communication terminal device capable of inputting and outputting data. The motion state monitoring device 4 may also include a server computer. In this embodiment, the motion state monitoring device 4 will be described as a tablet terminal.

[0035] The motion status monitoring device 4 is used by the user during, before and after the exercise test. The motion status monitoring device 4 accepts the selection of the motion to be monitored from the user and notifies the user of the attachment position 20 corresponding to the target body part. Then, the motion status monitoring device 4 transmits a request to start or stop measurement to the sensor 200 in response to the start or end of the exercise test.

[0036] Furthermore, in response to receiving the sensing information from the sensor 200, the operation state monitoring device 4 outputs sensing related information as a measurement result. Here, the sensing related information indicates information related to the sensing information, and may include the sensing information itself. Furthermore, the sensing related information may be information obtained by performing various conversion processes on the sensing information. Furthermore, the sensing related information may be information obtained by performing arithmetic processes based on the sensing information. The sensing related information may include information related to the above-mentioned operation state. Note that the information related to the above-mentioned operation state may include the sensing related information. In other words, the information related to the above-mentioned operation state may be information based on the sensing related information, or may include the sensing related information itself.

[0037] The operation state monitoring device 4 may be communicably connected to an external server (not shown) via a network. The external server may be a computer device or a cloud server on the Internet. In this case, the operation state monitoring device 4 may transmit sensing-related information or information related to the operation state of the subject P that it holds to the external server.

[0038] 2 and 3 are configuration diagrams illustrating the attachment of a sensor 200 to a measuring device 2 in an operating state monitoring system 1 according to the first embodiment. As shown in Fig. 2, the measuring device 2 has a sensor 200, an attachment pad 210, and a band 220. The band 220 is formed so as to be able to be wrapped around a target part of a subject P. The sensor 200 is incorporated in, for example, the attachment pad 210. The attachment pad 210 in which the sensor 200 is incorporated is formed so as to be able to be attached to and detached from the band 220.

[0039] 3, the band 220 is wrapped around the right upper arm, which is one of the target sites of the subject P. The sensor 200 is attached to the band 220 via the attachment pad 210 after pairing, calibration, and the like are completed.

[0040] FIG. 4 is a block diagram illustrating the measuring instrument 2 and the operating state monitoring device 4 in the operating state monitoring system 1 according to the first embodiment. As described above, the operating state monitoring system 1 includes the measuring instrument 2 and the operating state monitoring device 4. The measuring instrument 2 has a sensor 200. In the figure, the sensor 200 is the sensor 200 associated with the attachment position 20 selected based on the monitoring target operation from among the prepared sensors 200-1 to 200-11. The sensor 200 is assumed to have been paired with the operating state monitoring device 4 in advance and calibrated. The number of sensors 200 is not limited to one, and may be two or more.

[0041] The operation state monitoring device 4 includes an acquisition unit 41, a calculation processing unit 42, an image processing unit 43, a display unit 44, a display control unit 45, and a reception unit 46. The acquisition unit 41, the calculation processing unit 42, the image processing unit 43, the display unit 44, the display control unit 45, and the reception unit 46 have functions as an acquisition means, a calculation processing means, an image processing means, a display means, a display control means, and a reception means.

[0042] The acquiring unit 41 acquires the detection result by the sensor 200. Specifically, the acquiring unit 41 acquires sensing information from the sensor 200 attached to the target body part. The acquiring unit 41 also acquires the subject's captured image. For example, the acquiring unit 41 acquires the subject's captured image from the imaging unit 3. The acquiring unit 41 may acquire the subject's captured image from a storage device that stores the subject's captured image.

[0043] The acquisition unit 41 may also acquire an image depicting the motion state of the target body part. The image depicting the motion state of the target body part is called a subject-drawn image. The subject-drawn image may be simply called a drawn image. The image processing unit 43 may generate the subject-drawn image from the subject captured image. The acquisition unit 41 acquires the subject-drawn image from the image processing unit 43. The acquisition unit 41 may acquire the subject-drawn image from a storage device that stores the subject-drawn image. The acquisition unit 41 also acquires information input by the user via the reception unit 46.

[0044] The calculation processing unit 42 generates a calculation result representing the motion state of the target body part of the subject P based on the detection result by the sensor 200. The detection result by the sensor 200 includes the angle of the joint of the body of the subject P detected in the monitored motion, and the angle of the joint in an arbitrary coordinate system measured based on the detection result of any of the sensors. Hereinafter, the generation of the calculation result representing the motion state of the monitored motion is also referred to as the measurement of the monitored motion.

[0045] For example, the arithmetic processing unit 42 performs arithmetic processing based on the detection results of the sensor 200-1 attached to the right upper arm (attachment position 20-1) and the sensor 200-2 attached to the right forearm (attachment position 20-2) of the subject P, among the sensors 200-1 to 200-11. As a result, the arithmetic processing unit 42 generates a calculation result that indicates the motion state of the right elbow flexion and extension motion of the subject P.

[0046] Alternatively, the arithmetic processing unit 42 performs arithmetic processing based on the detection results of the sensor 200-5 attached to the waist (attachment position 20-5) and the sensor 200-8 attached to the right thigh (attachment position 20-8) of the subject P, among the sensors 200-1 to 200-11. As a result, the arithmetic processing unit 42 generates a calculation result that represents the motion state of the right side waist lateral bending motion of the subject P.

[0047] The calculation processing unit 42 may perform the calculation process using a trained model generated by machine learning using past detection results of the sensor 200. By performing the calculation process using such a trained model, the calculation processing unit 42 can more accurately calculate whether or not the motion state of the monitored motion of the subject P is good.

[0048] The calculation processing unit 42 may perform a process of visualizing the calculation results in a graph, table, or the like. This allows the user to easily understand the operating state of the target body part. Therefore, for example, this can be useful for assisting the subject P.

[0049] The image processing unit 43 processes the subject captured image. The image processing unit 43 also processes the subject drawn image. The image processing unit 43 processes at least one of the subject captured image and the subject drawn image.

[0050] The image processing unit 43 may generate a subject-drawn image of a 3D model that depicts the motion state of the subject P. The image processing unit 43 may generate a subject-drawn image of a 3D model that depicts the motion state captured in the subject-captured image. The image processing unit 43 may also generate a subject-drawn image of a 3D model that depicts a reference motion state in advance. The reference motion state includes an ideal motion state of the target part of the subject P. The image processing unit 43 may generate a subject-drawn image of an avatar instead of a 3D model. The avatar may include an animation image. In this way, the subject-drawn image is not necessarily limited to a 3D model, and may include a drawn image of an avatar.

[0051] The image processing unit 43 may generate a trajectory of the target part in the subject-drawn image of the 3D model. For example, the image processing unit 43 may generate a restriction of the target part in the subject-drawn image of the 3D model that depicts a motion state captured in the subject-captured image. The image processing unit 43 may also generate a trajectory of the target part in the subject-captured image. The image processing unit 43 may also generate a trajectory of the target part in the subject-drawn image of the 3D model that depicts a previously generated reference motion state.

[0052] The image processing unit 43 may generate a Lissajous figure by combining a plurality of trajectories. Specifically, for example, the image processing unit 43 generates, as a first trajectory, a trajectory of a target part in a subject-drawn image of a 3D model in which a motion state captured in a subject-captured image is drawn. Also, the image processing unit 43 generates, as a second image, a trajectory of a target part in a subject-drawn image of a 3D model in which a motion state that is a reference generated in advance is drawn. Then, the image processing unit 43 generates a Lissajous figure by combining the first trajectory and the second trajectory.

[0053] The image processing unit 43 may generate a figure indicating the movable area of ​​the target part in the subject-drawn image of the 3D model.

[0054] The display unit 44 displays the calculation results of the motion state of the target part generated by the calculation processing unit 42. The display unit 44 also displays at least one of the subject captured images and the subject drawn images processed by the image processing unit 43. For example, the display unit 44 displays the subject drawn image of the 3D model generated by the image processing unit 43. The display unit 44 may display the subject drawn image of the 3D model by enlarging or reducing it, or may display the subject captured image by enlarging or reducing it.

[0055] Display unit 44 displays the subject-drawn image including the trajectory of the target part generated by image processing unit 43. Display unit 44 may display the subject-captured image including the trajectory of the target part generated by image processing unit 43. Display unit 44 displays at least one of the subject-captured image and the subject-drawn image in synchronization with the calculation result.

[0056] The display unit 44 may display the Lissajous figure generated by the image processing unit 43. The display unit 44 may also display a figure indicating the generated movable area. The display unit 44 may display the Lissajous figure, the figure indicating the movable area, and the calculation results in a synchronized manner.

[0057] The display unit 44 may display at least one of the vital data and the walking speed. The display unit 44 may also display a plurality of subject captured images and subject drawn images at different times. The display unit 44 displays the calculation results, the subject captured images, the subject drawn images, etc. under the control of the display control unit 45.

[0058] The display control unit 45 controls the display of the display unit 44. Specifically, the display control unit 45 controls the display unit 44 to display the calculation results generated by the calculation processing unit 42, the subject-captured images and subject-drawn images that have been image-processed by the image processing unit 43, and the like.

[0059] The accepting unit 46 accepts information input by the user.

[0060] Next, an explanation will be given of an operation state monitoring method using the operation state monitoring system 1. Fig. 5 is a flow chart illustrating an example of a measurement method in the operation state monitoring method using the operation state monitoring system 1 according to the first embodiment. Fig. 6 is a flow chart illustrating an example of a display method in the operation state monitoring method using the operation state monitoring system 1 according to the first embodiment.

[0061] 5, in the operation status monitoring system 1, the measuring instrument 2 is associated with the mounting position 20. This allows pairing processing to be performed between the operation status monitoring device 4 and the measuring instrument 2.

[0062] Next, as shown in step S12, calibration of the sensor 200 in the measuring device 2 is performed. Calibration is, for example, a process of measuring an output value (error component) of the sensor 200 used to measure the monitored operation in a stationary state, and subtracting the error component from an actual measurement value. Here, it is assumed that the output value of the sensor 200 is stabilized after a predetermined period (about 20 seconds) has elapsed since the sensor 200 was made stationary. In this case, it is desirable that the output value of the sensor 200 after the predetermined period has elapsed since the sensor 200 was made stationary is used as the error component in the calibration. Therefore, in this embodiment, the output value of the sensor 200 after the predetermined period has elapsed since the user gave an instruction to start calibration after the sensor 200 was made stationary is used as the error component. In addition, "during calibration" means a processing period until the error component is determined, and "completed calibration" means that the output value (error component) of the sensor in a stationary state has been determined.

[0063] During calibration, for example, "calibration in progress" is displayed on the display unit 44. When calibration is completed, for example, "calibration completed" is displayed on the display unit 44. Note that the fact that calibration is in progress or that calibration has been completed does not necessarily have to be displayed on the display unit 44, and may be notified by other notification methods, such as by audio.

[0064] Next, as shown in step S13, the measuring device 2 is attached to the subject P. In this embodiment, among the sensors 200-1 to 200-11, the sensor 200-1, the sensor 200-2, the sensor 200-6, and the sensor 200-7 are attached to the right upper arm (attachment position 20-1), the right forearm (attachment position 20-2), the left upper arm (attachment position 20-6), and the left forearm (attachment position 20-7) of the subject P, respectively. The sensor 200 may be attached via an attachment pad 210 and a band 220.

[0065] Next, as shown in step S14, among the plurality of monitoring target movements, the monitoring target movements that can be measured using the sensor 200 attached to the subject P are measured.

[0066] Next, as shown in step S15, the motion state monitor 4 starts measuring the motion state of the target part.

[0067] Next, a display method will be described. Display by the display unit 44 of this embodiment may be performed during measurement in the exercise test, or may be performed before the start of measurement in the exercise test or after the end of measurement.

[0068] 6, based on the detection result by the sensor that detects the motion state of the target part, the calculation processing unit 42 generates a calculation result representing the motion state. The generated calculation result representing the motion state includes, for example, the measurement result of the monitored motion.

[0069] Next, as shown in step S22, the image processing unit 43 performs image processing on at least one of a subject captured image in which the motion state of the target part is captured and a subject drawn image in which the motion state is drawn.

[0070] Next, as shown in step S23, the display unit 44 displays the processed image and the calculation results in synchronization with each other.

[0071] 7 to 12 are diagrams illustrating examples of displays on the display unit 44 in the motion state monitoring system 1 according to the embodiment 1. A display image 300-1 before the start of measurement in an exercise test is shown in Fig. 7. As shown in Fig. 7, the display image 300-1 includes display areas 302, 304, 305, 306, 309, and 310.

[0072] Display area 302 displays icon images representing a plurality of mounting positions 20 that are candidates for mounting sensor 200. In display area 302, mounting positions 20 (positions indicated by "1", "2", "6", and "7" in the figure) that correspond to the selected measurement operation may be highlighted. This allows the user to easily visually identify mounting positions 20, thereby enabling the exercise test to be carried out smoothly.

[0073] In the display area 304, the rotation angles of the sensors 200-1, 200-2, ..., 200-11 linked to the mounting positions 20-1, 20-2, ..., 20-11 are displayed two-dimensionally. The rotation angles displayed here change dynamically in response to the movement of the sensor 200 linked to the movement of the subject P. Therefore, the user can identify the sensor 200 whose power is turned off or that is not operating normally via the display area 304 before starting measurement.

[0074] In the case where a plurality of sensors 200 are used in an exercise test, an input operation button for collectively calibrating the plurality of sensors 200 is displayed in the display area 305. This allows the user to easily request calibration for each of the plurality of sensors 200 via the display area 305.

[0075] Display area 306 displays an input operation button for starting an exercise test, i.e., for starting measurement by sensor 200. This allows the user to easily request, via display area 306, for measurement by sensor 200 to be started.

[0076] Sensing related information for each sensor 200 used is displayed in a display area 309. Before the start of measurement, the sensing related information is not yet displayed. A motion status indicator of the target part for each monitored motion that has been performed is displayed in a display area 310. Before the start of measurement, the motion status indicator of the target part is not yet displayed.

[0077] 8 shows a display image 300-2 at the end of measurement in the exercise test. Like display image 300-1, display image 300-2 includes display areas 302, 304, 305, 306, 309, and 310. Display areas 302 and 304 of display image 300-2 are similar to display areas 302 and 304 of display image 300-1 shown in FIG.

[0078] Display area 308 displays an input operation button for ending the exercise test, i.e., for stopping measurement by sensor 200. This allows the user to easily request, via display area 308, that measurement by sensor 200 be stopped.

[0079] The display area 309 displays sensing-related information for each sensor 200 used. Among the sensors 200-1, 200-2, 200-6, and 200-7 used, X based on the output of some of the sensors 200-1 and 200-6 is displayed. S ,Y S and Z S The rotation angle around the axis is displayed in time series.

[0080] The display area 310 displays the motion status index of the target part for each performed motion to be monitored. The motion status index is an index indicating the motion status of the target part when the motion to be monitored is performed. The calculation processing unit 42 calculates the motion status index of the target part based on the sensing related information of the sensor 200. For example, when the motion to be monitored is "right elbow flexion and extension", the sensing related information of the sensors 200-1 and 200-2 at the attachment positions 20-1 and 20-2 is used. In this case, the calculation processing unit 42 may calculate the motion status index based on the difference between the sensing related information of the sensors 200-1 and 200-2. The display area 310 displays the time series motion status index of some of the performed motions to be monitored.

[0081] FIG. 9 shows a display image 300-3. The display image 300-3 includes display areas 311 to 314. The display area 311 displays a subject captured image. The subject captured image may include a moving image. The display area 312 displays a subject-drawn image. The subject-drawn image displayed in the display area 312 includes a subject-drawn image of a 3D model generated by the image processing unit 43. The subject-drawn image of the 3D model may be generated by drawing an action state captured in the subject captured image. The display area 313 displays a calculation result representing an action state generated by the calculation processing unit 42 based on the detection result by the sensor 200. The calculation result includes, for example, a graph showing a change over time in a rotation angle.

[0082] In this manner, the display unit 44 may display the subject captured image and the subject drawn image. The display unit 44 may display a drawn image of a 3D model as the subject drawn image. The display unit 44 may display the subject captured image and the subject drawn image in a synchronized manner. In other words, the image capturing time of the subject captured image may be synchronized with the image capturing time of the subject captured image used to generate the subject drawn image. This allows the user to observe the motion state of the target part in correspondence with the 3D model. This makes it easier to understand the motion state.

[0083] Display unit 44 may enlarge or reduce the subject captured image in display area 311 and display it, or may enlarge or reduce the subject-drawn image in display area 312. Display unit 44 may enlarge or reduce the subject-drawn image of a 3D model and display it. This makes it easier to understand the operating state.

[0084] In the display area 312, a graphic 312a is displayed which indicates the movable area of ​​the target part in the subject-drawn image of the 3D model generated by the image processing unit 43. In addition, in the display area 314, vital data and walking speed are displayed. In this manner, the display unit 44 may display the graphic 312a which indicates the generated movable area, and may further display at least one of the vital data and walking speed. By using such a display method, the motion state can be grasped in more detail.

[0085] 10 shows a display image 300-4. The display image 300-4 includes display areas 311, 313, and 315. In the display area 315, similar to the display area 313, calculation results indicating the operating state generated by the calculation processing unit 42 are displayed. In the display area 315, a plurality of calculation results are combined and displayed in an enlarged manner.

[0086] In this manner, the display unit 44 may display the subject captured image and the calculation result. The display unit 44 may display the subject-drawn image and the calculation result instead of or in addition to the subject captured image. The display unit 44 may display at least one of the subject captured image and the subject-drawn image in synchronization with the calculation result. In other words, the imaging time at which the subject captured image is captured may be displayed in synchronization with the measurement time measured by the sensor 200. This allows the user to observe the motion state of the target part in correspondence with the 3D model, making it easier to understand the motion state.

[0087] FIG. 11 shows a display image 300-5. The display image 300-5 includes display areas 316 to 319. In the display area 316, a subject captured image at a predetermined imaging time is displayed. For example, a subject captured image including a motion state when the motion state of a target part is not good is displayed. In the display area 317, a calculation result is displayed in synchronization with the subject captured image in the display area 316. In the display area 318, a subject captured image at an imaging time different from the predetermined imaging time is displayed. For example, a subject captured image after the motion state has recovered to a good state is displayed. In the display area 319, a calculation result is displayed in synchronization with the subject captured image in the display area 318.

[0088] In this manner, the display unit 44 may display a plurality of subject-captured images taken at different times. Note that the display unit 44 may display a subject-drawn image obtained by drawing a plurality of subject-captured images taken at different times, or may display subject-captured images and subject-drawn images taken at different times.

[0089] FIG. 12 shows a display image 300-6. The display image 300-6 includes display areas 320 to 324. A subject-drawn image of a 3D model generated by the image processing unit 43 is displayed in the display area 320. The subject-drawn image in the display area 320 includes a trajectory 320a of a target part. The subject-drawn image displayed in the display area 320 may be a subject-drawn image of a 3D model that depicts a motion state captured in a captured image. The calculation result is displayed in the display area 321 in synchronization with the subject-drawn image in the display area 320.

[0090] A subject-drawn image of another 3D model is displayed in display area 322. The subject-drawn image in display area 322 includes a trajectory 322a of a target part. The subject-drawn image displayed in display area 322 may be a subject-drawn image of a 3D model that depicts a reference motion state generated in advance by image processing unit 43. The calculation result is displayed in display area 323 in synchronization with the subject-drawn image in display area 320.

[0091] In this way, the display unit 44 may display a subject-drawn image including the trajectories 320a and 322a of the target part in the subject-drawn image of the 3D model. In this case, the display unit 44 may display a subject-drawn image in which a motion state captured in the captured image is drawn as the subject-drawn image including the trajectories 320a and 322a, or may display a subject-drawn image of the 3D model in which a reference motion state generated in advance is drawn. The display unit 44 may display a subject-captured image including the trajectory of the target part in the captured image, which is the trajectory generated by the image processing unit 43.

[0092] A Lissajous figure generated by the image processing unit 43 is displayed in the display area 324. The Lissajous figure is generated by combining a first trajectory and a second trajectory. The first trajectory is, for example, a trajectory 320a of the target part as displayed in the display area 320. The second trajectory is, for example, a trajectory 322a of the target part as displayed in the display area 322.

[0093] In this way, the display unit 44 may display a Lissajous figure obtained by combining the first trajectory and the second trajectory generated by the image processing unit 43. By displaying the Lissajous figure, it is possible to display the difference between a reference operating state generated in advance and an operating state at the time of imaging, making it easier to understand the operating state.

[0094] Next, the effects of this embodiment will be described. The motion state monitoring system 1 of this embodiment synchronizes and displays at least one of the subject captured image and the subject drawn image with the calculation result. This makes it easier to understand the motion state of the target part.

[0095] The display unit 44 displays a 3D model of the subject's drawn image, making it easier to see the motion state. For example, the display unit 44 can enlarge or reduce the subject's drawn image before displaying it. This allows the user to focus on each target body part, making it easier to understand the motion state. The display unit 44 also displays a figure showing the movable area of ​​the target body part, vital data, walking speed, and the like, allowing the user to grasp the motion state in more detail.

[0096] The display unit 44 displays the trajectory of the target part in the motion state and the Lissajous figure, which allows the motion state of the target part to be understood in more detail.

[0097] In the above embodiment, the present disclosure has been described as a hardware configuration, but the present disclosure is not limited to this. The present disclosure can also be realized by having a processor execute a computer program, for example, an operation status monitoring program, to perform each process related to the operation status monitoring method.

[0098] In the above-described embodiment, the computer is configured as a computer system including a personal computer, a word processor, etc. However, the present invention is not limited to this, and the computer can also be configured as a LAN server, a computer (personal computer) communication host, a computer system connected to the Internet, etc. It is also possible to distribute functions to each device on the network and configure a computer using the entire network.

[0099] Fig. 13 is a schematic diagram illustrating an example of an operation state monitoring device 4 including a computer according to an embodiment. As shown in Fig. 13, the operation state monitoring device 4 may further include a processor PRC, a memory MMR, a storage device STR, and a user interface UI. The storage device STR stores the processes executed by each component of the operation state monitoring device 4 as a program. The processor PRC also loads the program from the storage device STR into the memory MMR and executes the program. In this way, the processor PRC realizes the functions of each component in the operation state monitoring device 4. The user interface UI may include input devices such as a keyboard, a mouse, and an imaging device, and output devices such as a display, a printer, and a speaker.

[0100] Each of the components of the operation state monitoring device 4 may be realized by dedicated hardware. Also, some or all of the components may be realized by a general-purpose or dedicated circuit, a processor PRC, or a combination of these. These may be configured by a single chip, or by multiple chips connected via a bus. Some or all of the components may be realized by a combination of the above-mentioned circuits, etc., and a program. Also, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-programmable Gate Array), a quantum processor (quantum computer control chip), etc. may be used as the processor PRC.

[0101] Furthermore, when some or all of the components of the operation status monitoring device 4 are realized by a plurality of information processing devices, circuits, etc., the plurality of information processing devices, circuits, etc. may be centrally arranged or distributed. For example, the information processing devices, circuits, etc. may be realized in a form in which each is connected via a communication network NW by a client-server system, a cloud computing system, etc. Also, the functions of the operation status monitoring device 4 may be provided in a SaaS (Software as a Service) format.

[0102] The order of execution of each process in the apparatus and method shown in the claims, specification, and drawings is not specifically stated as "before," "prior to," etc., and may be realized in any order as long as the output of a previous process is not used in a subsequent process. Even if the operational flow in the claims, specification, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the process in this order. [Explanation of symbols]

[0103] 1 Operation status monitoring system 2, 2-1, 2-2, 2-6, 2-7 Measuring instruments 3. Imaging unit 4. Operation status monitoring device 20, 20-1, 20-2, 20-3, 20-4, 20-5, 20-6 Installation position 20-7, 20-8, 20-9, 20-10, 20-11 Installation position 41 Acquisition Department 42 Processing unit 43 Image Processing Section 44 Display section 45 Display control section 46 Reception 200, 200-1, 200-2, 200-6, 200-7 Sensors 210 Mounting Pad 220 Band 300-1, 300-2, 300-3, 300-4, 300-5, 300-6 Display image 302, 304, 305, 306, 309, 310, 311, 312 display area 312a Shape 313, 314, 315, 316, 317, 318, 319, 320 display area 320a, 322a trajectory 321, 322, 323, 324 display area MMR Memory P Subject PRC Processor STR storage UI User Interface

Claims

1. A motion status monitoring system for monitoring a motion status of a target part of a subject's body, comprising: a calculation processing unit that generates a calculation result representing the operating state based on a detection result by a sensor that detects the operating state; an image processing unit that performs image processing on at least one of a captured image obtained by capturing the operational state and a drawn image obtained by drawing the operational state; a display unit that displays the image that has been subjected to image processing and the calculation result in a synchronized manner; An operating condition monitoring system equipped with

2. the image processing unit generates the drawn image of a 3D model that depicts the motion state or the drawn image of an avatar that depicts the motion state; The display unit displays the generated rendered image of the 3D model or the avatar. The operating condition monitoring system according to claim 1 .

3. the image processing unit generates a trajectory of the target part in the rendered image of the 3D model or the avatar; The display unit displays the drawn image including the generated trajectory. The operating condition monitoring system according to claim 2 .

4. The image processing unit generates a trajectory of the target part in the captured image, The display unit displays the captured image including the generated trajectory. The operating condition monitoring system according to claim 2 .

5. the image processing unit generates a trajectory of the target part in the drawn image of the 3D model or the avatar that depicts the motion state captured in the captured image, The display unit displays the drawn image including the generated trajectory. The operating condition monitoring system according to claim 2 .

6. the image processing unit generates a trajectory of the target part in the drawn image of the 3D model or the avatar that depicts the motion state that is a reference that has been generated in advance; The display unit displays the drawn image including the generated trajectory. The operating condition monitoring system according to claim 2 .

7. The image processing unit includes: generating a first trajectory of the target part in the drawn image of the 3D model or the avatar that depicts the motion state captured in the captured image; generating a second trajectory of the target part in the rendered image of the 3D model or the avatar that depicts the motion state that is a reference that has been generated in advance; A Lissajous figure is generated by combining the first trajectory and the second trajectory; The display unit displays the generated Lissajous figure. The operating condition monitoring system according to claim 2 .

8. The display unit displays the drawn image of the 3D model or the avatar by enlarging or reducing it. The operating condition monitoring system according to claim 2 .

9. the image processing unit generates a graphic representing a movable area of ​​the target part in the rendered image of the 3D model or the avatar; The display unit displays the graphic indicating the generated movable area. The operating condition monitoring system according to claim 2 .

10. The display unit further displays at least one of vital data and walking speed. The operating condition monitoring system according to claim 1 .

11. The display unit displays a plurality of the images at different times. The operating condition monitoring system according to claim 1 .

12. A motion status monitoring method for monitoring a motion status of a target part of a body of a subject using a motion status monitoring system, comprising: The operation status monitoring system includes: a calculation processing unit that generates a calculation result representing the operating state based on a detection result by a sensor that detects the operating state; an image processing unit that performs image processing on at least one of a captured image obtained by capturing the operational state and a drawn image obtained by drawing the operational state; a display unit that displays the image that has been subjected to image processing and the calculation result in a synchronized manner; having generating the calculation result representing the operating state based on the detection result; performing image processing on the image; displaying the image that has been subjected to image processing in synchronization with the calculation result; An operating condition monitoring method comprising:

13. A motion status monitoring program for causing a computer included in a motion status monitoring system to monitor a motion status of a target part of a subject's body, The operation status monitoring system includes: a calculation processing unit that generates a calculation result representing the operating state based on a detection result by a sensor that detects the operating state; an image processing unit that performs image processing on at least one of a captured image obtained by capturing the operational state and a drawn image obtained by drawing the operational state; a display unit that displays the image that has been subjected to image processing and the calculation result in a synchronized manner; having generating the calculation result representing the operating state based on the detection result; performing image processing on the image; displaying the image that has been subjected to image processing in synchronization with the calculation result; An operating status monitoring program that causes the computer to execute the above.

Images