Operating status monitoring system, operating status monitoring method, and operating status monitoring program

Abstract

To provide an operating state monitoring system, an operating state monitoring method and an operating state monitoring program capable of making an operating state in an object part of a subject's body easy to understand.SOLUTION: An operating state monitoring system is an operating state monitoring system 1 for monitoring an operating state in an object part of the body of a subject P and comprises: an arithmetic processing part 42 for generating an arithmetic result on the basis of a detection result by a sensor 200 for detecting the operating state; an image processing part 43 for performing image processing on at least one of a taken image of the operating state and a drawn image of the operating state; and a display part 44 for synchronously displaying the processed image and the arithmetic result.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to an operating state monitoring system, an operating state monitoring method, and an operating state monitoring program.

Background Art

[0002] Patent Document 1 discloses an operating state monitoring system that monitors the operating state of a target part of a subject's body. The display unit in the operating state monitoring system of Patent Document 1 displays an icon indicating the position of a sensor that detects the operating state, and a line graph showing the operating state detected by the sensor as an angular display from a predetermined position.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] With only the angular display by the line graph, it is difficult to understand the operating state of the target part of the subject's body when looking back at the data. It is desired to make it easier to understand the operating state of the target part of the subject.

[0005] The present disclosure has been made in view of the above background, and an object thereof is to provide an operating state monitoring system, an operating state monitoring method, and an operating state monitoring program that can make it easier to understand the operating state of a target part of a subject's body when looking back at the data.

Means for Solving the Problems

[0006] An action state monitoring system according to one embodiment of the present disclosure is an action state monitoring system for monitoring the action state of a target part of a subject's body, comprising: an arithmetic processing unit that generates a calculation result representing the action state based on detection results from a sensor that detects the action state; an image processing unit that performs image processing on at least one of an image captured of the action state and a drawing image depicting the action state; and a display unit that synchronizes and displays the image processed image and the calculation result. The arithmetic processing unit may perform arithmetic processing using a trained model generated by machine learning using past sensor detection results.

[0007] In the above operating state monitoring system, the image processing unit generates a rendering image of a 3D model depicting the operating state or a rendering image of an avatar depicting the operating state, and the display unit may display the generated rendering image of the 3D model.

[0008] In the above operating state monitoring system, the image processing unit may generate the trajectory of the target part in the drawn image of the 3D model or avatar, and the display unit may display the drawn image including the generated trajectory.

[0009] In the above operating state monitoring system, the image processing unit may generate a trajectory of the target part in the captured image, and the display unit may display the captured image including the generated trajectory.

[0010] In the above operating state monitoring system, the image processing unit generates a trajectory of the target part in the drawn image of the 3D model or avatar that depicts the operating state captured in the captured image, and the display unit may display the drawn image including the generated trajectory.

[0011] In the above operating state monitoring system, the image processing unit generates a trajectory of the target part in the drawn image of the 3D model or avatar that depicts the previously generated reference operating state, and the display unit may display the drawn image including the generated trajectory.

[0012] In the above-described operating state monitoring system, the image processing unit generates a first trajectory of the target part in the drawn image of the 3D model or avatar depicting the operating state captured in the captured image, generates a second trajectory of the target part in the drawn image of the 3D model or avatar depicting a pre-generated reference operating state, generates a Lissajous figure by combining the first and second trajectories, and the display unit may display the generated Lissajous figure.

[0013] In the above-described operating status monitoring system, the display unit may enlarge or reduce the rendering image of the 3D model or avatar to display it.

[0014] In the above operating state monitoring system, the image processing unit may generate a figure indicating the movable area of ​​the target part in the drawn image of the 3D model or avatar, and the display unit may display the generated figure indicating the movable area.

[0015] In the above operating status monitoring system, the display unit may further display at least one of vital data and walking speed.

[0016] In the above operating status monitoring system, the display unit may display multiple images taken at different times.

[0017] A method for monitoring the operating state according to one aspect of the present disclosure is a method for monitoring the operating state of a target part of a subject's body using an operating state monitoring system, wherein the operating state monitoring system comprises: an arithmetic processing unit that generates an arithmetic result representing the operating state based on a detection result from a sensor that detects the operating state; an image processing unit that performs image processing on at least one of an image captured of the operating state and an image drawn of the operating state; and a display unit that synchronizes and displays the image processed image and the arithmetic result, and further comprises the steps of: generating the arithmetic result representing the operating state based on the detection result; performing image processing on the image; and displaying the image processed image and the arithmetic result in synchronization.

[0018] An action state monitoring program according to one aspect of the present disclosure is an action state monitoring program that causes a computer included in an action state monitoring system to perform an action state monitoring of a target body part of a subject, wherein the action state monitoring system includes: an arithmetic processing unit that generates an arithmetic result representing the action state based on a detection result from a sensor that detects the action state; an image processing unit that performs image processing on at least one of an image captured of the action state and an image drawn of the action state; and a display unit that synchronizes and displays the image processed image and the arithmetic result, wherein the computer is caused to perform the steps of generating the arithmetic result representing the action state based on the detection result, performing image processing on the image, and synchronizing and displaying the image processed image and the arithmetic result. [Effects of the Invention]

[0019] According to this disclosure, it is possible to provide a motion state monitoring system, a motion state monitoring method, and a motion state monitoring program that make it easier to understand the motion state of a target part of a subject's body when reviewing the data. [Brief explanation of the drawing]

[0020] [Figure 1]It is a configuration diagram illustrating an operation state monitoring system according to Embodiment 1. [Figure 2] In the operation state monitoring system according to Embodiment 1, it is a configuration diagram illustrating the attachment of sensors of a measuring instrument. [Figure 3] In the operation state monitoring system according to Embodiment 1, it is a configuration diagram illustrating the attachment of sensors of a measuring instrument. [Figure 4] In the operation state monitoring system according to Embodiment 1, it is a block diagram illustrating a measuring instrument and an operation state monitoring device. [Figure 5] In the operation state monitoring method using the operation state monitoring system according to Embodiment 1, it is a flowchart diagram illustrating a measurement method. [Figure 6] In the operation state monitoring method using the operation state monitoring system according to Embodiment 1, it is a flowchart diagram illustrating a display method. [Figure 7] In the operation state monitoring system according to Embodiment 1, it is a diagram illustrating the display of a display unit. [Figure 8] In the operation state monitoring system according to Embodiment 1, it is a diagram illustrating the display of a display unit. [Figure 9] In the operation state monitoring system according to Embodiment 1, it is a diagram illustrating the display of a display unit. [Figure 10] In the operation state monitoring system according to Embodiment 1, it is a diagram illustrating the display of a display unit. [Figure 11] In the operation state monitoring system according to Embodiment 1, it is a diagram illustrating the display of a display unit. [Figure 12] In the operation state monitoring system according to Embodiment 1, it is a diagram illustrating the display of a display unit. [Figure 13] It is a schematic configuration diagram illustrating an operation state monitoring device including a computer according to an embodiment.

Modes for Carrying Out the Invention

[0021] The present disclosure will be described below through embodiments, but this disclosure is not limited to the embodiments described below. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means of solving the problem. For clarity of explanation, the following descriptions and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations have been omitted where necessary.

[0022] <Embodiment 1> The first embodiment of the motion state monitoring system 1 will be described. Figure 1 is a configuration diagram illustrating the motion state monitoring system according to the first embodiment. The motion state monitoring system 1 monitors the motion state of body parts of subject P. The motion state monitoring system 1 also provides support to bring subject P's movements closer to desired movements based on the monitoring results of subject P's movements. Specifically, the motion state monitoring system 1 measures the motor function of subject P, including rehabilitation trainees and elderly people. The motion state monitoring system 1 then supports subject P's training by analyzing, evaluating, and managing the measurement results. The motion state monitoring system 1 displays the analyzed, evaluated, and managed measurement results on a display unit. Subject P performs a motor test by attaching sensors 200 to predetermined parts of their body. For example, a motor test is a motor function test that measures motor function by measuring the motion state of body parts when subject P performs a specified movement. The motion state monitoring system 1 includes, for example, a training support system.

[0023] In the following, the specified movements are referred to as monitored movements. Monitored movements are defined in relation to body parts. For example, monitored movements include shoulder flexion and extension, shoulder abduction and adduction, shoulder internal and external rotation, neck flexion and extension, neck rotation, elbow flexion and extension, hip internal and external rotation, forearm pronation and supination, and thoracolumbar lateral flexion. If a body part is on either the left or right side, the monitored movements may be defined separately for the left and right sides. The body part being monitored is called the target body part. One or more target body parts may be associated with a single monitored movement, and the same target body part may be associated with different monitored movements.

[0024] As shown in Figure 1, the motion state monitoring system 1 comprises a measuring instrument 2 and a motion state monitoring device 4. The motion state monitoring device 4 may also be referred to as the motion state monitoring system 1. The motion state monitoring device 4 is a device that monitors the motion state of a target part of the subject P's body. The motion state of a target part of the subject P's body will be referred to below as the motion state of the target part. The motion state monitoring system 1 may also include an imaging unit 3. The measuring instruments 2-1, 2-2, ... etc shown in the figure will be collectively referred to as measuring instrument 2.

[0025] Measuring instrument 2 measures the operating state of the target part. Specifically, measuring instrument 2 is a measuring device that measures the direction and amount of movement of the target part. Measuring instrument 2 has a sensor 200 that detects the operating state. In this embodiment, measuring instrument 2 may also have an acceleration sensor and an angular velocity sensor. Measuring instrument 2 measures its own acceleration and angular velocity. Specifically, measuring instrument 2 may include a 3-axis acceleration sensor and a 3-axis angular velocity sensor. In this case, measuring instrument 2 measures the amount of movement in the three axes of the XYZ axis and the rotation angle around the three axes. Note that the number of measurement axes is not limited to three axes, but may be two or fewer. Measuring instrument 2 may also have a geomagnetic sensor that detects the Earth's magnetic field and measures the direction it is facing. Note that the sensors 200-1, 200-2, ... etc shown in the figure are collectively referred to as sensor 200.

[0026] Each measuring instrument 2 is connected to the operating status monitoring device 4 in a communication-enabled state. In this embodiment, communication between each measuring instrument 2 and the operating status monitoring device 4 is via short-range wireless communication such as Bluetooth®, NFC (Near Field Communication), and ZigBee. However, this communication is not limited to these and may also be wireless communication via a network such as a wireless LAN (Local Area Network). Alternatively, this communication may also be wired communication via a network including the Internet, LAN, WAN (Wide Area Network), or a combination thereof.

[0027] The measuring instrument 2 has a sensor 200 and a mounting mechanism for the sensor 200. The sensor 200 is attached to mounting positions 20 corresponding to target body parts of subject P via the mounting mechanism. In order to measure various monitored target movements, each of the multiple sensors 200 is linked to each target body part of subject P and can be attached to the linked target body part. In the figure, the target body parts to which the sensors can be attached are shown as mounting positions 20-1, 20-2, ..., 20-11. Each mounting position 20-1, 20-2, ..., 20-11 is linked to each sensor 200-1, 200-2, ..., 200-11. For example, mounting positions 20-1, 20-2, ..., 20-11 are called the right upper arm, right forearm, head, chest (trunk), waist (pelvis), left upper arm, left forearm, right thigh, right lower leg, left thigh, and left lower leg, respectively. The linking of mounting positions 20 to sensors 200 is done by pairing sensors 200 with the operating status monitoring device 4 in advance, and then associating the identification information (ID) of mounting positions 20 with the ID of sensors 200 on the application of the operating status monitoring device 4. Note that mounting positions 20-1, 20-2, ..., etc. are collectively referred to as mounting positions 20.

[0028] In this embodiment, the mounting positions 20 used in the exercise test are selected from mounting positions 20-1 to 20-11 according to the monitored movement selected by the user. The user is the user of the motion state monitoring device 4, for example, the subject P themselves or the staff member conducting the exercise test. The subject P or staff member then attaches the sensors 200 (sensors 200-1, 200-2, 200-6, 200-7 in this figure) linked to the selected mounting positions 20 (mounting positions 20-1, 20-2, 20-6, 20-7 in this figure) to the subject P's body and starts the exercise test.

[0029] Although it is stated that each system will have multiple sensors 200 associated with each of the multiple mounting positions 20, the number of mounting positions 20 may be as few as one. Similarly, the number of sensors 200 may be as few as one.

[0030] Sensor 200 starts measuring when the motion test begins and transmits sensing information to the motion state monitoring device 4. The sensing information may include acceleration information, angular velocity information, or quaternion information. The sensing information may also include components in each of these measurement axis directions (X-axis direction, Y-axis direction, Z-axis direction). Sensor 200 then stops measuring when the motion test is completed.

[0031] The imaging unit 3 captures images of the operating state of the target area. The imaging unit 3 outputs the captured images to the operating state monitoring device 4. The imaging unit 3 includes, for example, a camera. The imaging unit 3 may capture video or still images. Therefore, the images may include video or still images. Images capturing the operating state of the target area of ​​subject P's body are called subject-captured images. Subject-captured images are sometimes simply referred to as captured images.

[0032] The imaging unit 3 is connected to the operating status monitoring device 4 in a communication-enabled manner. In this embodiment, communication between the imaging unit 3 and the operating status monitoring device 4 may be selected from the various wireless and wired communication methods described in the section on communication between the measuring instrument 2 and the operating status monitoring device 4.

[0033] Note that the imaging unit 3 is not an essential component of the operating state monitoring system 1. For example, if the operating state monitoring device 4 can acquire the subject's captured images, the imaging unit 3 may not be necessary. For example, the subject's captured images may be stored in a predetermined storage device (for example, the storage device STR in Figure 13), and the operating state monitoring device 4 may acquire the subject's captured images from the storage device.

[0034] The motion state monitoring device 4 monitors the motion state of the target body part during the exercise test. The motion state monitoring device 4 analyzes, evaluates, and manages information related to the motion state. The motion state monitoring device 4 includes, for example, a computer device. Specifically, the motion state monitoring device 4 may include a personal computer, a notebook computer, a mobile phone, a smartphone, a tablet terminal, or other communication terminal device capable of data input / output. The motion state monitoring device 4 may also include a server computer. In this embodiment, the motion state monitoring device 4 will be described as a tablet terminal.

[0035] The motion status monitoring device 4 is used by the user during and before / after the exercise test. The motion status monitoring device 4 receives a selection of the movement to be monitored from the user and notifies the user of the mounting position 20 corresponding to the target body part. Then, depending on whether the exercise test has started or ended, the motion status monitoring device 4 sends a request to the sensor 200 to start or stop measurement.

[0036] Furthermore, the operating state monitoring device 4 outputs sensing-related information as a measurement result in response to receiving sensing information from the sensor 200. Here, sensing-related information refers to information related to the sensing information and may include the sensing information itself. Sensing-related information may also be information obtained by applying various conversion processes to the sensing information. Moreover, sensing-related information may also be information obtained by performing calculations based on the sensing information. Sensing-related information may also include the information regarding the operating state described above. Note that the information regarding the operating state described above may include sensing-related information. In other words, the information regarding the operating state described above may be information based on sensing-related information, or it may include sensing-related information itself.

[0037] The operating status monitoring device 4 may be connected to an external server (not shown) via a network so as to be able to communicate with it. The external server may be a computer device or a cloud server on the internet. In this case, the operating status monitoring device 4 may transmit sensing-related information or information regarding the operating status of subject P that it holds to the external server.

[0038] Figures 2 and 3 are configuration diagrams illustrating the mounting of the sensor 200 of the measuring instrument 2 in the operating state monitoring system 1 according to Embodiment 1. As shown in Figure 2, the measuring instrument 2 includes a sensor 200, a mounting pad 210, and a band 220. The band 220 is formed to be wrapped around a target area of ​​the subject P. The sensor 200 is incorporated, for example, into the mounting pad 210. The mounting pad 210, into which the sensor 200 is incorporated, is formed to be detachably attached to the band 220.

[0039] As shown in Figure 3, the band 220 is wrapped around the upper right arm, which is one of the target areas for subject P. After pairing and calibration are completed, the sensor 200 is attached to the band 220 via the mounting pad 210.

[0040] Figure 4 is a block diagram illustrating the measuring instrument 2 and the operating state monitoring device 4 in the operating state monitoring system 1 according to Embodiment 1. As described above, the operating state monitoring system 1 includes the measuring instrument 2 and the operating state monitoring device 4. The measuring instrument 2 has a sensor 200. In the figure, the sensor 200 is the sensor 200 that is linked to the mounting position 20 selected from the available sensors 200-1 to 200-11 based on the operation to be monitored. The sensor 200 is assumed to have been paired with the operating state monitoring device 4 and calibrated in advance. Note that the number of sensors 200 is not limited to one, but may be two or more.

[0041] The operating status monitoring device 4 comprises an acquisition unit 41, a calculation processing unit 42, an image processing unit 43, a display unit 44, a display control unit 45, and a reception unit 46. The acquisition unit 41, calculation processing unit 42, image processing unit 43, display unit 44, display control unit 45, and reception unit 46 have the functions of acquisition means, calculation processing means, image processing means, display means, display control means, and reception means.

[0042] The acquisition unit 41 acquires detection results from the sensor 200. Specifically, the acquisition unit 41 acquires sensing information from the sensor 200 attached to the target area. The acquisition unit 41 also acquires subject images. For example, the acquisition unit 41 acquires subject images from the imaging unit 3. Alternatively, the acquisition unit 41 may acquire subject images from a storage device that stores subject images.

[0043] The acquisition unit 41 may also acquire an image depicting the operating state of the target area. The image depicting the operating state of the target area is called the subject drawing image. The subject drawing image may also be simply called the drawing image. The image processing unit 43 may generate the subject drawing image from the subject image. The acquisition unit 41 acquires the subject drawing image from the image processing unit 43. The acquisition unit 41 may also acquire the subject drawing image from a storage device that stores the subject drawing image. The acquisition unit 41 also acquires information input from the user by the reception unit 46.

[0044] The arithmetic processing unit 42 generates calculation results representing the movement state of a target part of the subject P's body based on the detection results from the sensor 200. The detection results from the sensor 200 include the angles of the joints of the subject P's body detected during the monitored movement, as well as the angles of the joints in an arbitrary coordinate system measured based on the detection results of any of the sensors. Hereinafter, the generation of calculation results representing the movement state of the monitored movement will also be referred to as the measurement of the monitored movement.

[0045] For example, the arithmetic processing unit 42 performs calculations based on the detection results of sensor 200-1, which is attached to the upper right arm (mounting position 20-1) of subject P, and sensor 200-2, which is attached to the right forearm (mounting position 20-2), among sensors 200-1 to 200-11. As a result, the arithmetic processing unit 42 generates calculation results that represent the motion state of subject P's right elbow flexion and extension movement.

[0046] Alternatively, the arithmetic processing unit 42 performs calculations based on the detection results of sensor 200-5, which is attached to the lower back (mounting position 20-5) of subject P, and sensor 200-8, which is attached to the right thigh (mounting position 20-8), from among sensors 200-1 to 200-11. As a result, the arithmetic processing unit 42 generates calculation results that represent the motion state of the lateral flexion movement of the right side of subject P's lower back.

[0047] Furthermore, the arithmetic processing unit 42 may perform calculations using a trained model generated by machine learning using past detection results of the sensor 200. By performing calculations using such a trained model, the arithmetic processing unit 42 can more accurately calculate whether the operating state of the monitored operation of subject P is good or not.

[0048] The calculation processing unit 42 may also perform processing to visualize the calculation results in a graph, table, or the like. This makes it easier for the user to understand the operating state of the target body part. Therefore, it can be used, for example, to assist subject P.

[0049] The image processing unit 43 processes the subject's captured images. The image processing unit 43 also processes the subject's drawn images. The image processing unit 43 performs image processing on at least one of the subject's captured images and the subject's drawn images.

[0050] The image processing unit 43 may generate a subject drawing image of a 3D model depicting the operating state of subject P. The image processing unit 43 may generate a subject drawing image of a 3D model depicting the operating state captured in the subject image. Alternatively, the image processing unit 43 may pre-generate a subject drawing image of a 3D model depicting a reference operating state. The reference operating state includes the ideal operating state of the target part of subject P. The image processing unit 43 may generate a subject drawing image of an avatar instead of a 3D model. The avatar may include an animated image. Thus, the subject drawing image is not necessarily limited to a 3D model and may include an avatar drawing image.

[0051] The image processing unit 43 may generate the trajectory of the target area in the 3D model subject rendering image. For example, the image processing unit 43 may generate the trajectory of the target area in the 3D model subject rendering image, which depicts the motion state captured in the subject image. Alternatively, the image processing unit 43 may generate the trajectory of the target area in the subject image. Alternatively, the image processing unit 43 may generate the trajectory of the target area in the 3D model subject rendering image, which depicts a pre-generated reference motion state.

[0052] The image processing unit 43 may generate a Lissajous figure by combining multiple trajectories. Specifically, for example, the image processing unit 43 generates the trajectory of the target part in the subject drawing image of a 3D model that depicts the motion state captured in the subject's image as a first trajectory. The image processing unit 43 also generates the trajectory of the target part in the subject drawing image of a 3D model that depicts a pre-generated reference motion state as a second image. Then, the image processing unit 43 generates a Lissajous figure by combining the first trajectory and the second trajectory.

[0053] The image processing unit 43 may generate a figure indicating the range of motion of the target part in the 3D model of the subject.

[0054] The display unit 44 displays the calculation results of the operating state of the target part generated by the calculation processing unit 42. The display unit 44 also displays at least one of the subject's captured image and subject's drawn image processed by the image processing unit 43. For example, the display unit 44 displays a subject's drawn image of a 3D model generated by the image processing unit 43. The display unit 44 may enlarge or reduce the subject's drawn image of the 3D model, or enlarge or reduce the subject's captured image.

[0055] The display unit 44 displays a subject drawing image that includes the trajectory of the target area generated by the image processing unit 43. The display unit 44 may also display a subject image that includes the trajectory of the target area generated by the image processing unit 43. The display unit 44 displays at least one of the subject image and the subject drawing image in synchronization with the calculation result.

[0056] The display unit 44 may display a Lissajous figure generated by the image processing unit 43. The display unit 44 may also display a figure indicating the generated movable area. The display unit 44 may also display the Lissajous figure, the figure indicating the movable area, and the calculation results in a synchronized manner.

[0057] The display unit 44 may display at least one of vital data and walking speed. The display unit 44 may also display multiple subject images and subject drawings taken at different times. The display unit 44 displays the calculation results, as well as the subject images and subject drawings, etc., under the control of the display control unit 45.

[0058] The display control unit 45 controls the display of the display unit 44. Specifically, the display control unit 45 controls the display of the calculation results generated by the calculation processing unit 42, as well as the subject image captured and subject drawing images processed by the image processing unit 43, on the display unit 44.

[0059] The reception unit 46 receives information entered by the user.

[0060] Next, a method for monitoring the operating status using the operating status monitoring system 1 will be described. Figure 5 is a flowchart illustrating an example of the measurement method in the operating status monitoring method using the operating status monitoring system 1 according to Embodiment 1. Figure 6 is a flowchart illustrating an example of the display method in the operating status monitoring method using the operating status monitoring system 1 according to Embodiment 1.

[0061] As shown in step S11 of Figure 5, the operating state monitoring system 1 performs a correspondence between the measuring instrument 2 and the mounting position 20. This enables pairing processing between the operating state monitoring device 4 and the measuring instrument 2.

[0062] Next, as shown in step S12, the sensor 200 in the measuring instrument 2 is calibrated. Calibration is a process in which the output value (error component) of the sensor 200, which is used to measure the operation of the monitored object, is measured in a stationary state, and this error component is subtracted from the measured value. Here, it is assumed that the output value of the sensor 200 stabilizes after a predetermined period (approximately 20 seconds) has elapsed since the sensor 200 was made stationary. In that case, it is desirable that the output value of the sensor 200 after a predetermined period has elapsed since the sensor 200 was made stationary be used as the error component in the calibration. Therefore, in this embodiment, the output value of the sensor 200 after a predetermined period has elapsed since the user gave an instruction to start calibration after the sensor 200 was made stationary is used as the error component. Furthermore, "during calibration" means the processing period until the error component is determined, and "calibration complete" means that the output value (error component) of the sensor in a stationary state has been determined.

[0063] During calibration, the display unit 44 will display, for example, "Calibration in progress." Once calibration is complete, the display unit 44 will display, for example, "Calibration complete." Note that the status of calibration and the completion of calibration are not limited to being displayed on the display unit 44; notification may also be given by other means, such as voice notification.

[0064] Next, as shown in step S13, the measuring instrument 2 is attached to the subject P. In this embodiment, of sensors 200-1 to 200-11, sensors 200-1, 200-2, 200-6, and 200-7 are attached to the upper right arm (mounting position 20-1), right forearm (mounting position 20-2), left upper arm (mounting position 20-6), and left forearm (mounting position 20-7) of subject P, respectively. The sensors 200 may be attached via mounting pads 210 and bands 220.

[0065] Next, as shown in step S14, among the multiple monitored actions, the monitored actions that can be measured using the sensor 200 attached to subject P are measured.

[0066] Next, as shown in step S15, the operating state monitoring device 4 starts measuring the operating state of the target part.

[0067] Next, the display method will be described. The display by the display unit 44 of this embodiment may be performed during measurement in the exercise test, or it may be performed before the start of measurement and after the end of measurement in the exercise test.

[0068] As shown in step S21 of Figure 6, the arithmetic processing unit 42 generates a calculation result representing the operating state based on the detection result from the sensor that detects the operating state of the target part. The generated calculation result representing the operating state includes, for example, the measurement result of the monitored operation.

[0069] Next, as shown in step S22, the image processing unit 43 performs image processing on at least one of the subject image captured, which captures the operating state of the target area, and the subject drawing image, which depicts the operating state.

[0070] Next, as shown in step S23, the display unit 44 displays the processed image and the calculation result in synchronization.

[0071] Figures 7 to 12 illustrate the display of the display unit 44 in the operating state monitoring system 1 according to Embodiment 1. Figure 7 shows the display image 300-1 before the start of measurement in an exercise test. As shown in Figure 7, the display image 300-1 includes display areas 302, 304, 305, 306, 309 and 310.

[0072] The display area 302 displays icon images representing multiple mounting positions 20 that are candidates for mounting the sensor 200. In the display area 302, the mounting position 20 corresponding to the selected measurement operation (the positions indicated as "1", "2", "6", and "7" in the figure) may be highlighted. This allows the user to easily see the mounting position 20, thereby enabling smooth execution of the motion test.

[0073] The display area 304 shows the rotation angles of each sensor 200-1, 200-2, ..., 200-11 associated with each mounting position 20-1, 20-2, ..., 20-11 in two dimensions. The rotation angles displayed here change dynamically in accordance with the movement of the sensors 200, which is linked to the movement of the subject P. Therefore, the user can identify sensors 200 that are powered off or not functioning properly via the display area 304 before starting the measurement.

[0074] The display area 305 shows input buttons for calibrating multiple sensors 200 simultaneously when multiple sensors 200 are used in a motion test. This allows the user to easily request calibration for each of the multiple sensors 200 via the display area 305.

[0075] The display area 306 shows an input button to start the exercise test, that is, to start measurement by the sensor 200. This allows the user to easily request that measurement by the sensor 200 be started via the display area 306.

[0076] Display area 309 displays sensing-related information for each sensor 200 used. Before measurement starts, sensing-related information is not yet displayed. Display area 310 displays operational status indicators for each monitored operation performed on the target part. Before measurement starts, operational status indicators for the target part are not yet displayed.

[0077] Figure 8 shows display image 300-2 at the end of the measurement in the exercise test. Display image 300-2 includes display areas 302, 304, 305, 306, 309, and 310, similar to display image 300-1. Display areas 302 and 304 of display image 300-2 are the same as display areas 302 and 304 of display image 300-1 shown in Figure 6.

[0078] The display area 308 shows an input button to end the exercise test, that is, to stop the measurement by the sensor 200. This allows the user to easily request that the measurement by the sensor 200 be stopped via the display area 308.

[0079] The display area 309 shows sensing-related information for each sensor 200 used. Of the sensors 200-1, 200-2, 200-6, and 200-7 used, X based on the output of some sensors 200-1 and 200-6. S ,Y S and Z S The rotation angle around the axis is displayed in chronological order.

[0080] The display area 310 displays the operational status index of the target part for each monitored operation that has been performed. The operational status index is an indicator that shows the operational state of the target part when the monitored operation is performed. The arithmetic processing unit 42 calculates the operational status index of the target part based on the sensing-related information of the sensor 200. For example, if the monitored operation was "right elbow flexion and extension", the sensing-related information of sensors 200-1 and 200-2 at mounting positions 20-1 and 20-2 is used. In this case, the arithmetic processing unit 42 may calculate the operational status index based on the difference in the sensing-related information of sensors 200-1 and 200-2. The display area 310 displays the time-series operational status index for some of the monitored operations that have been performed.

[0081] Figure 9 shows display image 300-3. Display image 300-3 includes display areas 311 to 314. Display area 311 displays the subject's captured image. The subject's captured image may include video. Display area 312 displays the subject's drawn image. The subject's drawn image displayed in display area 312 includes a 3D model of the subject's drawn image generated by the image processing unit 43. The 3D model of the subject's drawn image may be generated by drawing the motion state captured on the subject's captured image. Display area 313 displays the calculation results representing the motion state generated by the arithmetic processing unit 42 based on the detection results from the sensor 200. The calculation results include, for example, a graph showing the time change of the rotation angle.

[0082] Thus, the display unit 44 may display the subject image and the subject drawing image. The display unit 44 may also display a drawing of a 3D model as the subject drawing image. The display unit 44 may also display the subject image and the subject drawing image in sync. That is, the imaging time when the subject image was captured and the imaging time when the subject image used to generate the subject drawing image may be synchronized and displayed. This allows the user to observe the operating state of the target area in correspondence with the 3D model. Therefore, the operating state can be easily understood.

[0083] The display unit 44 may enlarge or reduce the subject image captured in the display area 311, or enlarge or reduce the subject drawing image in the display area 312. The display unit 44 may also enlarge or reduce the subject drawing image of the 3D model. This makes it easier to understand the operating status.

[0084] Display area 312 displays a figure 312a indicating the range of motion of the target part in the 3D model of the subject drawn by the image processing unit 43. Display area 314 displays vital data and walking speed. Thus, the display unit 44 may display the figure 312a indicating the generated range of motion, or it may further display at least one of the vital data and walking speed. This display method allows for a more detailed understanding of the operating state.

[0085] Figure 10 shows display image 300-4. Display image 300-4 includes display areas 311, 313, and 315. Display area 315, like display area 313, displays the calculation results representing the operating state generated by the arithmetic processing unit 42. In display area 315, multiple calculation results are combined and displayed in an enlarged view.

[0086] Thus, the display unit 44 may display the subject's captured image and the calculation results. Alternatively, the display unit 44 may display the subject's drawn image and the calculation results in addition to, or instead of, the subject's captured image. The display unit 44 may also display the calculation results in synchronization with at least one of the subject's captured image and the subject's drawn image. In other words, the imaging time when the subject's captured image was taken and the measurement time measured by the sensor 200 may be displayed in synchronization. This allows the user to observe the operating state of the target area in correspondence with a 3D model, making it easier to understand the operating state.

[0087] Figure 11 shows display image 300-5. Display image 300-5 includes display areas 316 to 319. Display area 316 displays the subject image captured at a predetermined imaging time. For example, it displays the subject image including the operating state when the operating state of the target area is not good. Display area 317 displays the calculation results synchronized with the subject image captured in display area 316. Display area 318 displays the subject image captured at an imaging time different from the predetermined imaging time. For example, it displays the subject image captured after the operating state has recovered to good condition. Display area 319 displays the calculation results synchronized with the subject image captured in display area 318.

[0088] Thus, the display unit 44 may display multiple subject images captured at different times. The display unit 44 may also display a subject drawing image obtained by drawing multiple subject images captured at different times, or it may display subject images captured at different times and subject drawing images.

[0089] Figure 12 shows display image 300-6. Display image 300-6 includes display areas 320-324. Display area 320 displays a subject drawing image of a 3D model generated by the image processing unit 43. The subject drawing image in display area 320 includes the trajectory 320a of the target area. The subject drawing image displayed in display area 320 may also be a subject drawing image of a 3D model in which the motion state captured in the image is drawn. Display area 321 displays the calculation results in synchronization with the subject drawing image in display area 320.

[0090] Display area 322 displays a subject drawing image of another 3D model. The subject drawing image in display area 322 includes the trajectory 322a of the target area. The subject drawing image displayed in display area 322 may also be a subject drawing image of a 3D model that has been pre-generated by the image processing unit 43 to depict a reference operating state. Display area 323 displays the calculation results in synchronization with the subject drawing image in display area 320.

[0091] Thus, the display unit 44 may display a subject drawing image that includes the trajectories 320a and 322a of the target area in the subject drawing image of the 3D model. In this case, the display unit 44 may display a subject drawing image that depicts the motion state captured in the captured image, or it may display a subject drawing image of a 3D model that depicts a pre-generated reference motion state, as the subject drawing image that includes the trajectories 320a and 322a. The display unit 44 may also display a subject captured image that includes the trajectories of the target area in the captured image, which are generated by the image processing unit 43.

[0092] The display area 324 displays a Lissajous figure generated by the image processing unit 43. The Lissajous figure is generated by combining a first trajectory and a second trajectory. The first trajectory is, for example, the trajectory 320a of the target area, as shown in the display area 320. The second trajectory is, for example, the trajectory 322a of the target area, as shown in the display area 322.

[0093] Thus, the display unit 44 may display a Lissajous figure obtained by combining the first and second trajectories generated by the image processing unit 43. By displaying a Lissajous figure, the difference between a pre-generated reference operating state and the operating state at the time of imaging can be displayed, making it easier to understand the operating state.

[0094] Next, the effects of this embodiment will be described. The operating state monitoring system 1 of this embodiment synchronizes and displays at least one of the subject's captured image and subject's drawn image with the calculation result. Therefore, the operating state of the target body part can be easily understood.

[0095] The display unit 44 displays a 3D model of the subject, making it easier to see the subject's movements. For example, the display unit 44 can enlarge or reduce the image of the subject. This allows for focus on individual body parts and makes it easier to understand the subject's movements. Furthermore, the display unit 44 can display figures indicating the range of motion of the target body parts, vital data, walking speed, etc., allowing for a more detailed understanding of the subject's movements.

[0096] The display unit 44 displays the trajectory and Lissajous figure of the target part during operation. This allows for a more detailed understanding of the operation state of the target part.

[0097] In the embodiments described above, the disclosure was explained as a hardware configuration, but the disclosure is not limited thereto. The disclosure can also be implemented by having a processor execute a computer program, for example, an operating status monitoring program, for each process related to the operating status monitoring method.

[0098] In the embodiments described above, the computer is comprised of a computer system including a personal computer and a word processor. However, it is not limited to this; the computer can also be comprised of a LAN server, a computer (PC) communication host, a computer system connected to the Internet, and so on. Furthermore, it is possible to distribute functions among various devices on the network and constitute a computer across the entire network.

[0099] Figure 13 is a schematic diagram illustrating an operational status monitoring device 4 including a computer according to an embodiment. As shown in Figure 13, the operational status monitoring device 4 may further include a processor PRC, memory MMR, storage device STR, and a user interface UI. The storage device STR stores programs for the processes to be executed by each component of the operational status monitoring device 4. The processor PRC loads the programs from the storage device STR into the memory MMR and executes them. In this way, the processor PRC realizes the functions of each component in the operational status monitoring device 4. The user interface UI may include input devices such as a keyboard, mouse, and imaging device, and output devices such as a display, printer, and speaker.

[0100] Each component of the operating status monitoring device 4 may be implemented with dedicated hardware. Furthermore, some or all of each component may be implemented by general-purpose or dedicated circuits, processors (PRCs), etc., or combinations thereof. These may be implemented by a single chip or by multiple chips connected via a bus. Some or all of each component may be implemented by a combination of the aforementioned circuits, etc., and a program. Additionally, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-programmable Gate Array), quantum processor (quantum computer control chip), etc., can be used as the processor (PRC).

[0101] Furthermore, if some or all of the components of the operating status monitoring device 4 are implemented by multiple information processing devices or circuits, these devices may be centrally located or distributed. For example, the information processing devices or circuits may be implemented in a form where each is connected via a communication network (NW) by a client-server system, cloud computing system, etc. Also, the functions of the operating status monitoring device 4 may be provided in SaaS (Software as a Service) format.

[0102] The execution order of each process in the apparatus and method described in the claims, specification, and drawings is not explicitly stated as "before," "prior to," etc., and the processes can be implemented in any order unless the output of a previous process is used in a later process. Even if the operation flow in the claims, specification, and drawings is described using phrases such as "first," "next," etc. for convenience, it does not mean that it is essential to perform the operations in that order. [Explanation of symbols]

[0103] 1. Operating Status Monitoring System 2, 2-1, 2-2, 2-6, 2-7 Measuring Instruments 3. Imaging Unit 4. Operating status monitoring device Mounting positions: 20, 20-1, 20-2, 20-3, 20-4, 20-5, 20-6 Mounting positions 20-7, 20-8, 20-9, 20-10, 20-11 41 Acquisition Department 42 Arithmetic Processing Unit 43 Image Processing Unit 44 Display section 45 Display Control Unit 46 Reception Department 200, 200-1, 200-2, 200-6, 200-7 sensors 210 Mounting Pad 220 bands Display images for 300-1, 300-2, 300-3, 300-4, 300-5, and 300-6. 302, 304, 305, 306, 309, 310, 311, 312 display area 312a Figure 313, 314, 315, 316, 317, 318, 319, 320 display area 320a, 322a trajectory 321, 322, 323, 324 display area MMR memory P Subject PRC Processor STR storage UI User Interface

Claims

1. A motion state monitoring system that monitors the motion state of a target body part of a subject, A calculation processing unit that generates a calculation result representing the operating state based on the detection result from the sensor that detects the operating state, An image processing unit that performs image processing on at least one of the captured image and the drawn image depicting the operating state, A display unit that synchronizes and displays the image processed and the calculation result, Equipped with, The image processing unit generates the rendering image of the 3D model depicting the operating state or the rendering image of the avatar depicting the operating state. The display unit displays the generated 3D model or the drawn image of the avatar. A system for monitoring the operating status, The aforementioned image processing unit, A first trajectory of the target part in the drawn image of the 3D model or avatar, which depicts the motion state captured in the captured image, is generated. A second trajectory of the target part is generated in the drawn image of the 3D model or avatar that depicts the previously generated reference operating state. A Lissajous figure is generated by combining the first and second trajectories, The display unit displays the generated Lissajous figure. Operating status monitoring system.

2. The display unit displays the drawing image including the generated first trajectory. The operating status monitoring system according to claim 1.

3. The display unit displays the drawing image including the generated second trajectory. The operating status monitoring system according to claim 1.

4. The display unit displays the 3D model or the drawn image of the avatar by enlarging or reducing it. The operating status monitoring system according to claim 1.

5. The image processing unit generates a figure indicating the movable area of ​​the target part in the drawn image of the 3D model or the avatar. The display unit displays the figure indicating the generated movable area. The operating status monitoring system according to claim 1.

6. The display unit further displays at least one of vital data and walking speed. The operating status monitoring system according to claim 1.

7. The display unit displays multiple images at different times. The operating status monitoring system according to claim 1.

8. A motion state monitoring method for monitoring the motion state of a target body part of a subject using a motion state monitoring system, The aforementioned operating status monitoring system is A calculation processing unit that generates a calculation result representing the operating state based on the detection result from the sensor that detects the operating state, An image processing unit that performs image processing on at least one of the captured image and the drawn image depicting the operating state, A display unit that synchronizes and displays the image processed and the calculation result, It has, The steps include generating the calculation result representing the operating state based on the detection result, The steps include: performing image processing on the aforementioned image, The steps include: displaying the image processed by the image processing and the calculation result in synchronization; Equipped with, In the step of performing image processing on the aforementioned image, The image processing unit generates the rendering image of the 3D model depicting the operating state or the rendering image of the avatar depicting the operating state. In the step of displaying the image processed and the calculation result in synchronization, The display unit displays the generated 3D model or the drawn image of the avatar. A method for monitoring the operating status, In the step of performing image processing on the aforementioned image, The aforementioned image processing unit, A first trajectory of the target part in the drawn image of the 3D model or avatar, which depicts the motion state captured in the captured image, is generated. A second trajectory of the target part is generated in the drawn image of the 3D model or avatar that depicts the previously generated reference operating state. A Lissajous figure is generated by combining the first and second trajectories, In the step of displaying the image processed and the calculation result in synchronization, The display unit displays the generated Lissajous figure. Method for monitoring operating status.

9. A motion state monitoring program that causes a computer included in a motion state monitoring system to perform the operation of monitoring the motion state of a target body part of a subject, The aforementioned operating status monitoring system is A calculation processing unit that generates a calculation result representing the operating state based on the detection result from the sensor that detects the operating state, An image processing unit that performs image processing on at least one of the captured image and the drawn image depicting the operating state, A display unit that synchronizes and displays the image processed and the calculation result, It has, The steps include generating the calculation result representing the operating state based on the detection result, The steps include: performing image processing on the aforementioned image, The steps include: displaying the image processed by the image processing and the calculation result in synchronization; An operational status monitoring program that causes the computer to execute the following: In the step of performing image processing on the aforementioned image, The image processing unit generates the rendering image of the 3D model depicting the operating state or the rendering image of the avatar depicting the operating state. In the step of displaying the image processed and the calculation result in synchronization, The display unit displays the generated 3D model or the drawn image of the avatar. A status monitoring program that causes the computer to perform the following: In the step of performing image processing on the aforementioned image, The aforementioned image processing unit, A first trajectory of the target part in the drawn image of the 3D model or avatar, which depicts the motion state captured in the captured image, is generated. A second trajectory of the target part is generated in the drawn image of the 3D model or avatar that depicts the previously generated reference operating state. A Lissajous figure is generated by combining the first and second trajectories, In the step of displaying the image processed and the calculation result in synchronization, The display unit displays the generated Lissajous figure. A program that causes the computer to perform the following operation status monitoring.

Images