Operating status monitoring system, its control method, and control program

Abstract

To provide a motion state monitoring system that allows a sensor to be attached by allowing a sensor attaching position to be grasped smoothly, and to provide a control method and a control program thereof.SOLUTION: A motion state monitoring system that monitors a subject's motion according to a result of the detection from a plurality of sensors attached to each of a plurality of regions of the subject's body includes: a storage unit in which a motion to be monitored and a sensor attaching position are stored in association with each other; and a display control unit for displaying a sensor attaching position determined according to a designated motion to be monitored in a diagram simulating a human body by referring to the storage unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an operation state monitoring system, a control method thereof, and a control program.

Background Art

[0002] Patent Document 1 discloses an operation state monitoring system that monitors the operation state of a subject based on detection results from a plurality of sensors attached to respective ones of a plurality of parts of the subject's body.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In an operation state monitoring system such as related art, it is difficult to grasp the attachment position of a sensor corresponding to a measurement target.

[0005] The present disclosure has been made in view of the above background, and an object thereof is to provide an operation state monitoring system, a control method thereof, and a control program that enable the attachment position of a sensor to be smoothly grasped and the sensor to be attached.

Means for Solving the Problems

[0006] The operation state monitoring system according to the present disclosure is an operation state monitoring system that monitors the operation of a subject according to detection results from a plurality of sensors attached to respective ones of a plurality of parts of the subject's body, a storage unit in which a monitoring target operation and an attachment position of a sensor are stored in association with each other, A display control unit, which refers to the memory unit, displays the sensor mounting position determined according to the specified monitored operation on a diagram simulating a human body. It is equipped with.

[0007] The operating status monitoring system may further include a processing unit that associates the mounting position with the sensor based on the operation input for the displayed mounting position.

[0008] The aforementioned operation input may be an operation input in which a sensor icon corresponding to the sensor is dragged and dropped to the displayed mounting position.

[0009] The aforementioned motion state monitoring system may further include a designation unit that designates the monitored motion in response to operation inputs to joints in the diagram modeling the human body.

[0010] When a plurality of monitored operations, including a first monitored operation and a second monitored operation, are specified, the display control unit may display the first mounting position determined according to the first monitored operation and the second mounting position determined according to the second monitored operation in different display modes.

[0011] The display control unit may move the figure that mimics the human body based on the captured image of the subject.

[0012] The aforementioned motion state monitoring system may further include a designation unit that uses a machine learning model to extract motion characteristics from captured images of the subject and designates the motion corresponding to the characteristics as the motion to be monitored.

[0013] The operation status monitoring system may further include a designation unit that specifies the operation to be monitored based on the mounting position history of the sensor.

[0014] The control method for the operational status monitoring system relating to this disclosure is: A control method for a motion state monitoring system that monitors the movements of a subject in accordance with detection results from multiple sensors attached to multiple parts of the subject's body, The system references a memory unit that stores the monitored operation and the sensor mounting position in association, and displays the sensor mounting position determined according to the specified monitored operation on a diagram simulating the human body.

[0015] The control program relating to this disclosure is A control program that causes a computer to execute control processing in a motion state monitoring system that monitors the movements of a subject in accordance with detection results from multiple sensors attached to each of multiple parts of the subject's body, The system references a memory unit that stores the monitored actions and their corresponding sensor mounting positions, and then instructs the computer to display the sensor mounting position, determined according to the specified monitored action, on a diagram simulating the human body. [Effects of the Invention]

[0016] This disclosure provides an operational status monitoring system, a control method, and a control program that enable smooth determination of the sensor's mounting position and subsequent sensor installation. [Brief explanation of the drawing]

[0017] [Figure 1] This is a block diagram showing an example configuration of the operating status monitoring system according to Embodiment 1. [Figure 2] This figure shows an example of the mounting locations for measuring instruments attached to a subject. [Figure 3] This figure shows an example of the configuration of measuring instruments provided in the operating status monitoring system according to Embodiment 1. [Figure 4] Figure 3 shows an example of how to attach the measuring instrument shown in Figure 3 to the subject. [Figure 5] This figure shows an example of a display screen. [Figure 6] Figure 1 is a flowchart showing the operation of the operational status monitoring system.

Best Mode for Carrying Out the Invention

[0018] Hereinafter, the present invention will be described through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Also, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. For the sake of clarity of explanation, the following description and drawings have been appropriately omitted and simplified. In each drawing, the same reference numerals are assigned to the same elements, and redundant explanations are omitted as necessary.

[0019] <Embodiment 1> FIG. 1 is a block diagram showing a configuration example of an operation state monitoring system 1 according to Embodiment 1. The operation state monitoring system 1 is a system that monitors the operation state of a subject. Based on this monitoring result, a user such as an assistant can, for example, provide support to bring the subject's operation closer to a desired operation. This will be specifically described below.

[0020] As shown in FIG. 1, the operation state monitoring system 1 includes an operation state monitoring device 10 and a plurality of measuring instruments 20. The operation state monitoring device 10 can also be referred to as an operation state monitoring system by itself. The operation state monitoring device 10 and the plurality of measuring instruments 20 are configured to be able to communicate with each other via a wired or wireless network. Note that the operation state monitoring device 10 and the plurality of measuring instruments 20 may perform short-range wireless communication compliant with Bluetooth (registered trademark) standard, NFC (Near field communication), UWB (Ultra Wideband), WiFi (registered trademark), etc. In the present embodiment, the case where 11 measuring instruments 20 are provided will be described as an example. Hereinafter, each of the 20 measuring instruments 20 will also be referred to as measuring instruments 20_1 to 20_11 for distinction.

[0021] Each measuring instrument 20_1 to 20_11 is attached to a specific body part p1 to p11 of the subject P, and uses motion sensors (hereinafter simply referred to as sensors) 21_1 to 21_11, consisting of a gyro sensor and an accelerometer, to detect the movement of body parts p1 to p11. Each measuring instrument 20_1 to 20_11 is associated with body parts p1 to p11 through a pairing process performed with the motion state monitoring device 10.

[0022] Figure 2 shows an example of the attachment sites for measuring devices 20_1 to 20_11 on subject P's body. In the example in Figure 2, the attachment sites p1 to p11 for measuring devices 20_1 to 20_11 are the right upper arm, right forearm, head, chest (trunk), waist (pelvis), left upper arm, left forearm, right thigh, right lower leg, left thigh, and left lower leg, respectively. Note that it is not necessary for all measuring devices 20_1 to 20_11 to be attached to subject P's body. Only the measuring devices 20_1 to 20_11 necessary for measuring the target movements (including body part movements) that the user wishes to monitor should be attached to subject P's body.

[0023] The movements to be monitored include, for example, right shoulder flexion and extension, right shoulder abduction and adduction, right shoulder internal and external rotation, right elbow flexion and extension, right forearm pronation and supination, head flexion and extension, head rotation, thoracolumbar flexion and extension, thoracolumbar rotation, thoracolumbar lateral flexion, left shoulder flexion and extension, left shoulder abduction and adduction, left shoulder internal and external rotation, left elbow flexion and extension, and left forearm pronation and supination. In addition, the movements to be monitored also include the movement of the body part to which the sensor is attached.

[0024] For example, the monitored movements may include the angles of the joints of subject P's body, measured based on the detection results of multiple sensors, or the angles of the joints in an arbitrary coordinate system, measured based on the detection results of any of the sensors. The monitored movements may be specified by a user, such as a caregiver.

[0025] Here, as an example, we will monitor the flexion and extension of the right elbow of subject P. The flexion and extension of the right elbow can be measured based on the detection results of sensors attached to the right upper arm (part p1) and the right forearm (part p2). In this case, for example, two different measuring instruments are attached to subject P's right upper arm (part p1) and right forearm (part p2).

[0026] Furthermore, users may select multiple different monitored actions simultaneously. For example, a user may select "right elbow flexion / extension" and "right shoulder internal / external rotation," or "left elbow flexion / extension" and "left shoulder internal / external rotation."

[0027] Right elbow flexion and extension can be measured based on the detection results of sensors attached to the right upper arm (part p1) and right forearm (part p2). Similarly, right shoulder internal and external rotation can be measured based on the detection results of sensors attached to the right upper arm (part p1) and right forearm (part p2). In addition, left elbow flexion and extension can be measured based on the detection results of sensors attached to the left upper arm (part p6) and left forearm (part p7). Similarly, left shoulder internal and external rotation can be measured based on the detection results of sensors attached to the left upper arm (part p6) and left forearm (part p7).

[0028] (Example configuration of measuring instruments 20_1 to 20_11) Figure 3 shows an example of the configuration of measuring instrument 20_1. Note that the configurations of measuring instruments 20_2 to 20_11 are the same as those of measuring instrument 20_1, so their explanation is omitted.

[0029] As shown in Figure 3, the measuring instrument 20_1 includes a sensor 21_1, a mounting pad 22_1, and a belt 23_1. The belt 23_1 is formed to be wrapped around the area of ​​the subject P whose movement is to be detected. The sensor 21_1 is incorporated into, for example, the mounting pad 22_1. The mounting pad 22_1, into which the sensor 21_1 is incorporated, is formed to be detachably attached to the belt 23_1.

[0030] Figure 4 shows an example of how the measuring instrument 20_1 is attached to subject P. In the example in Figure 4, the belt 23_1 is wrapped around the upper right arm (part p1), which is one of the parts of subject P's body whose movement is to be detected. The sensor 21_1 is attached to the belt 23_1 via the mounting pad 22_1 after the mapping process and calibration are completed.

[0031] (Example configuration of the operating status monitoring device 10) The operating state monitoring device 10 is a device that outputs a calculation result representing the operating state of subject P based on the detection results (sensing values) of sensors 21_1 to 21_11. The operating state monitoring device 10 may be, for example, a PC (Personal Computer), a mobile phone terminal, a smartphone, or a tablet terminal. The operating state monitoring device 10 may be configured to communicate with sensors 21_1 to 21_11 via a network (not shown). The operating state monitoring device 10 can also be called an operating state monitoring system. The operating state monitoring device 10 may include a processor and a memory that stores various control programs. In this case, the operating state monitoring device 10 has the functionality of a computer and performs processing based on various control programs, etc.

[0032] As shown in Figure 1, the operating status monitoring device 10 comprises a storage unit 11, a display unit 12, a designation unit 13, a display control unit 14, a reception unit 15, and a processing unit 16.

[0033] The memory unit 11 is implemented using a computer-accessible storage device. The memory unit 11 stores the monitored operation and the sensor mounting position in association. For example, the flexion and extension of the right elbow is associated with the right upper arm (part p1) and the right forearm (part p2).

[0034] The display unit 12 is, for example, a display device that displays multiple sensor icons corresponding to multiple sensors 21_1 to 21_11. Figure 5 shows an example of the display screen S displayed on the display unit 12. The area of ​​the display screen S that displays the sensor icons is called the sensor icon display area S1. The display screen including the sensor icon display area S1 includes a schematic diagram of the human body S2 that shows the parts of the body to which the sensors are attached. The display screen S shown in Figure 5 is displayed when processing a one-to-one correspondence between any of the multiple sensors 21 and any of the parts of the subject P to which the sensors are attached.

[0035] In the example shown in Figure 5, the front and back sides of the human body are displayed separately. The attachment points p1 to p11 of subject P in Figure 2 correspond to p_1 to p_11 in the schematic human body diagram S2 in Figure 5, respectively. Hereafter, as needed, p_1 to p_11 in the schematic human body diagram S2 will be referred to as the right upper arm, right forearm, head, back (trunk), waist (pelvis), left upper arm, left forearm, right thigh, right lower leg, left thigh, and left lower leg, respectively. The display unit 12 can also display the calculation results based on the detection results of one or more sensors, for example, in a graph.

[0036] Referring to Figure 1, the designation unit 13 specifies the subject's actions to be monitored. The designation unit 13 may specify multiple actions to be monitored.

[0037] The designation unit 13 may specify the monitored operation in response to user input. For example, if the display screen S includes a selection list listing items of monitored operations (e.g., right shoulder flexion and extension, right shoulder abduction and adduction, etc.), the designation unit 13 may specify the monitored operation selected by the user.

[0038] Furthermore, the designation unit 13 may specify the monitored action in accordance with input operations on the joints in the human body diagram S2. Specifically, the designation unit 13 may specify the monitored action relating to the joint tapped by the user. For example, if the user taps the right elbow in the human body diagram S2, the designation unit 13 may specify right elbow flexion and extension as the monitored action. If there are multiple actions relating to the tapped joint, the designation unit 13 may further specify the monitored action based on input operations representing the direction of rotation around the joint, etc.

[0039] The designation unit 13 may also determine the operation to be monitored based on the sensor's mounting position history. The designation unit 13 may designate the operation associated with the previous mounting position as the operation to be monitored, or it may designate the operation associated with the mounting position where the sensor has been attached the most times as the operation to be monitored. Furthermore, if there are multiple operations related to the joints tapped by the user, the designation unit 13 may also designate the operation to be monitored taking into account the sensor's mounting position history.

[0040] The designation unit 13 may use a machine learning model to extract motion characteristics from the captured images of subject P and designate the motion corresponding to those characteristics as the motion to be monitored. Subject P can be designated as the motion to be monitored by performing an action similar to the motion to be monitored. Alternatively, the designation unit 13 may designate the motion to be monitored based on information entered by the user via voice or text. In this case, the designation unit 13 may use a large-scale language model.

[0041] The display control unit 14 refers to the storage unit 11 and displays the mounting part associated with the specified monitored operation on the display unit 12. The display control unit 14 may, for example, highlight the mounting part associated with the monitored operation using a different display method (e.g., color, blinking, shading, etc.) than other parts. If multiple monitored operations are specified, the display control unit 14 may display the mounting part associated with the first monitored operation (also referred to as the first part) and the mounting part associated with the second monitored operation (also referred to as the second part) using different display methods (e.g., blinking speed). The display control unit 14 may also make the thickness of the lines surrounding the first part and the thickness of the lines surrounding the second part different. Furthermore, the display control unit 14 may make the size of the displayed parts different.

[0042] For example, if right elbow flexion and extension is specified, p_1 corresponding to the upper right arm (part p1) and p_2 corresponding to the right forearm (part p2) will be highlighted, as shown in Figure 5.

[0043] The display control unit 14 may move the anatomical diagram S2 based on the captured image of subject P. The user can easily understand the relationship between the anatomical diagram S2 and subject P's body.

[0044] The multiple sensor icons shown above are selected by the user to associate the sensor 21 with the part p to be mounted. The reception unit 15 receives setting operations for the sensor icons displayed on the display unit 12. Setting operations are input operations for the sensor icons to associate the sensor corresponding to the sensor icon with one of the parts p to be mounted. For example, the user moves the sensor icon i21_1 displayed in the sensor icon display area S1 to the upper right arm part p_1 on the human body diagram S2. Specifically, the user drags and drops the sensor icon i21_1 displayed in the sensor icon display area S1 to the upper right arm part p_1 on the human body diagram S2 using mouse operation or touch operation. This setting operation is shown by the dotted line in Figure 5. In this way, the reception unit 15 receives the setting operation for the sensor icons by the user. Since p_1 and p_2 are highlighted, the user can perform the setting operation smoothly.

[0045] Referring to Figure 1, the processing unit 16 associates the sensor corresponding to the sensor icon with the mounting location on the subject P, according to the set operation. This association process is performed by pairing the operating state monitoring device 10 and the sensor 21 in advance, and linking the identification information of the mounting location p with the identification information of the sensor 21 on the application of the operating state monitoring device 10.

[0046] By displaying (e.g., highlighting) the mounting location corresponding to the monitored operation, the operation status monitoring system 1 according to Embodiment 1 enables smooth identification of the sensor mounting position and subsequent sensor installation.

[0047] Note that the setting operation is not limited to drag and drop. The setting operation can be any operation input for the displayed mounting position. The reception unit 15 may accept the setting operation when the user clicks the sensor icon displayed in the sensor icon display area S1 and the mounting target area in the human body schematic diagram S2 within a predetermined time.

[0048] Alternatively, the user may click on the target area for installation in the human body diagram S2, which may trigger the reception unit 15 to accept the setting operation. In response to this setting operation, the processing unit 16 can automatically associate one of the sensors corresponding to the multiple sensor icons displayed on the display unit 12 with the target area p of the subject P.

[0049] (Operation of the operational status monitoring device 10) Next, the operation of the motion state monitoring device 10 will be explained using Figure 6. Figure 6 is a flowchart showing the operation of the motion state monitoring device 10. Here, as described above, the motion state monitoring device 10 will monitor the flexion and extension of the subject P's right elbow. That is, sensors will be attached to the subject P's upper right arm (part p1) and right forearm (part p2). It will be assumed that the monitored motion and the sensor attachment positions are associated and stored in the memory unit 11.

[0050] First, the operating status monitoring device 10 displays multiple sensor icons i21_1 to i21_11, each corresponding to one of the multiple sensors, on the display unit 12 (S11).

[0051] The display unit 12 displays a screen that includes a sensor icon display area S1 and a schematic diagram of the human body S2 showing the mounting target area, as shown in Figure 5.

[0052] Next, the designation unit 13 of the operation status monitoring device 10 specifies the operation to be monitored (S12). The designation unit 13 may, for example, designate the flexion and extension of the right elbow, selected by the user, as the operation to be monitored.

[0053] The display unit 12 displays the mounting locations for sensors used to measure the specified monitored operation. In the human body diagram S2, the display unit 12 highlights the upper right arm p1 and the right forearm p2 with a different display method (color, flashing, shading, etc.) than the other parts p3 to p11. That is, in the human body diagram S2 of Figure 5, the upper right arm p1 and the right forearm p2 are displayed with a different method than the other parts p3 to p11.

[0054] Then, in the operating status monitoring device 10, the user, for example, drags and drops one sensor icon i21_1 from the sensor icons i21_1 to i21_11 displayed in the sensor icon display area S1 onto the upper right arm p_1 of the human body schematic diagram S2. As a result, the reception unit 15 accepts the setting operation (S13).

[0055] Subsequently, the operation status monitoring device 10 associates the identification information of sensor 21_1 corresponding to the sensor icon i21_1 with the identification information of the subject P's upper right arm (part p1) to which the sensor is to be attached, according to the set operation. This completes the process of associating sensor 21_1 with subject P's upper right arm (part p1) (S14). Similarly, for other attachment sites (right forearm (part p2)), drag-and-drop with sensor icon i21_2 and the process of associating sensor 21_2 with subject P's right forearm (part p2) are performed. In other words, steps S12 and S13 in Figure 6 can be repeated for each attachment site on the subject.

[0056] After the correspondence between sensor 21_1 and mounting location p1, and between sensor 21_2 and mounting location p2, the sensors 21_1 and 21_2 used to measure the monitored operation are then calibrated (S15). Calibration is a process that measures the output value (error component) of the sensor used to measure the monitored operation in a stationary state, and subtracts that error component from the measured value. In this example, at least sensors 21_1 and 21_2 are calibrated. However, calibration is not limited to sensors used to measure the monitored operation; for example, it may be performed on all sensors 21_1 to 21_11 before the display process of the sensor icon corresponding to the paired sensor.

[0057] After calibration is complete, sensors 21_1 and 21_2 are attached to subject P (S16). Subsequently, the monitored operation is measured based on the detection results of sensors 21_1 and 21_2 (S17).

[0058] The calculation result representing the "right elbow flexion and extension" motion state can be calculated from the difference between the detection result of sensor 21_1 attached to subject P's upper right arm (part p1) and the detection result of sensor 21_2 attached to the right forearm (part p2). The motion state monitoring device 10 generates a calculation result representing the "right elbow flexion and extension" motion state based on the detection results from sensors 21_1 and 21_2, respectively. The display unit 12 displays the details of the measurement results (for example, the measurement results as a graph).

[0059] As described above, the operating status monitoring system according to this embodiment displays the mounting location corresponding to the specified monitored operation on a schematic diagram of the human body. This makes it possible to easily understand the mounting position of the sensor and install the sensor.

[0060] Note that the processing order of the operating status monitoring system 1 is not limited to the order shown in Figure 6. For example, calibration may be performed before the display process of the sensor icon corresponding to the paired sensor.

[0061] If the monitored operation and the sensor mounting position are associated and stored in an external server or the like, the operation status monitoring device 10 may temporarily receive information from the external server or the like and store the received information in the storage unit 11 of the operation status monitoring device 10, which has a temporary storage function.

[0062] The operating status monitoring device 10 may be a PC, but it may also be a server, which is not necessarily a personal computer.

[0063] Furthermore, this disclosure can be implemented by having a CPU (Central Processing Unit) execute a computer program to perform some or all of the processing of the operational status monitoring system 1.

[0064] The program described above includes, when loaded into a computer, a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include RAM (Random-Access Memory), ROM (Read-Only Memory), flash memory, SSD (Solid-State Drive), or other memory technologies, CD-ROM, DVD (Digital Versatile Disc), Blu-ray® disc, or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals. [Explanation of symbols]

[0065] 1. Operating Status Monitoring System 10. Operating status monitoring device 11 Storage section 12 Display section 13 Specified part 14 Display Control Unit 15 Reception Department 20 Measuring Instruments 20_1~20_11 Measuring Instruments 21 sensors 21_1~21_11 Sensors 22_1 Pad 23_1 Belt P Subject p1~p11 site S display screen S1 Sensor icon display area S2 Human body schematic diagram

Claims

1. A motion state monitoring system that monitors the movements of a subject in accordance with detection results from multiple sensors attached to each of multiple parts of the subject's body, A memory unit that stores the monitored operation and the sensor's mounting position in correspondence, A display control unit, which refers to the memory unit, displays the sensor mounting position determined according to the specified monitored operation on a diagram simulating a human body. A processing unit that associates the mounting position with the sensor based on the operation input for the displayed mounting position. Equipped with, The aforementioned operation input is an operation input in which the sensor icon corresponding to the sensor is dragged and dropped to the displayed mounting position. The system includes a designation unit that uses a machine learning model to extract motion characteristics from the subject's captured images and designates the motion corresponding to those characteristics as the motion to be monitored. Operating status monitoring system.

2. A designation unit that specifies the monitored operation in response to the operation input to the joints in the diagram modeling the human body. The operating status monitoring system according to claim 1, further comprising:

3. When a plurality of monitored operations, including a first monitored operation and a second monitored operation, are specified, the display control unit displays the first mounting position determined according to the first monitored operation and the second mounting position determined according to the second monitored operation in different display modes. The operating status monitoring system according to claim 1 or 2.

4. The display control unit moves the figure that mimics the human body based on the captured image of the subject. The operating status monitoring system according to claim 1 or 2.

5. A control method for a motion state monitoring system that monitors the movements of a subject in accordance with detection results from multiple sensors attached to multiple parts of the subject's body, The process involves referring to a memory unit that stores the monitored operation and the sensor mounting position in association, and displaying the sensor mounting position determined according to the specified monitored operation on a diagram simulating the human body. A step of associating the mounting position with the sensor based on the operation input for the displayed mounting position. Includes, The aforementioned operation input is an operation input in which the sensor icon corresponding to the sensor is dragged and dropped to the displayed mounting position. The steps include: extracting motion characteristics from the subject's captured images using a machine learning model, and designating the motion corresponding to the characteristics as the motion to be monitored. A control method for an operating status monitoring system, including the following.

6. A control program that causes a computer to execute control processing in a motion state monitoring system that monitors the movements of a subject in accordance with detection results from multiple sensors attached to each of multiple parts of the subject's body, The process involves referring to a memory unit that stores the monitored operation and the sensor mounting position in association, and displaying the sensor mounting position determined according to the specified monitored operation on a diagram simulating the human body. Based on the operation input for the displayed mounting position, a process is performed to associate the mounting position with the sensor. Have the computer run it, The aforementioned operation input is an operation input in which the sensor icon corresponding to the sensor is dragged and dropped to the displayed mounting position. A process that extracts motion characteristics from the subject's captured images using a machine learning model, and designates the motion corresponding to the characteristics as the motion to be monitored. A control program that instructs a computer to execute a command.

Images