Judgment method, judgment device, judgment system, cross-reality device, learning method, learning device, program, and storage medium

Abstract

The objective is to provide a determination method, determination device, determination system, cross-reality device, program, and storage medium that can more easily determine the burden on a person, the risk to a person, or the skill level of a person. Alternatively, the objective is to provide a learning method, learning device, program, and storage medium that can train a model to estimate the force acting on a person's body using easily obtainable data. [Solution] The determination method according to the embodiment involves causing the processing device to measure the coordinates of a hand from an image of a person's hand. The determination method involves causing the processing device to receive detected values ​​from a sensor of a device held by a person. The determination method involves causing the processing device to input the coordinates and detected values ​​into an estimation model for estimating the force acting on the body. The determination method involves causing the processing device to use the force acting on the person's body output from the estimation model to determine one or more of the following from a group consisting of load on the person, danger to the person, and the person's skill level.

Description

Technical Field

[0001] Embodiments of the present invention relate to a determination method, a determination device, a determination system, a cross-reality device, a learning method, a learning device, a program, and a storage medium.

Background Art

[0002] When a person operates, there is a possibility that an excessive load is applied to the body, causing pain or injury to the body. Also, in order to improve the operation, it is effective to examine the proficiency of that person.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An embodiment of the present invention aims to provide a determination method, a determination device, a determination system, a cross-reality device, a program, and a storage medium that can more easily determine the load on a person, the risk to the person, or the proficiency of the person. Another embodiment of the present invention aims to provide a learning method, a learning device, a program, and a storage medium that train a model so as to be able to estimate the acting force on a person's body using easily acquirable data.

Means for Solving the Problems

[0005] The determination method according to the embodiment involves causing a processing device to measure the coordinates of a person's hand from an image of the hand. The determination method also involves causing the processing device to receive a detected value from a sensor of a device held by the person. The determination method causes the processing device to input the coordinates and the detected value into an estimation model for estimating the force acting on the body. The determination method causes the processing device to determine one or more of the following, selected from the group consisting of the load on the person, the danger to the person, and the person's skill level, using the force acting on the person's body output from the estimation model. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 is a flowchart showing the processing method according to an embodiment. [Figure 2] Figure 2 is a schematic diagram showing the configuration of the determination system according to the embodiment. [Figure 3] Figure 3 is a table illustrating a dataset. [Figure 4] Figure 4 is a table illustrating a dataset. [Figure 5] Figure 5 is a schematic diagram illustrating the learning method of the estimation model. [Figure 6] Figure 6 is a flowchart showing the learning method of the estimation model. [Figure 7] Figure 7 is a flowchart showing the learning method of the risk assessment model. [Figure 8] Figure 8 is a schematic diagram illustrating the learning method of the risk assessment model. [Figure 9] Figure 9 is a flowchart showing the learning method of the proficiency assessment model. [Figure 10] Figure 10 is a schematic diagram illustrating the learning method of the proficiency assessment model. [Figure 11] Figure 11 is a flowchart showing the determination method according to the embodiment. [Figure 12] Figure 12 is a table illustrating the output results from the first estimation model. [Figure 13]Figure 13 is a table illustrating thresholds for determining the load. [Figure 14] Figure 14 is a flowchart illustrating output control based on the load determination result. [Figure 15] Figure 15 is a schematic diagram showing an example of a cross-reality device according to an embodiment. [Figure 16] Figure 16 is a schematic diagram showing an example of output from the determination system according to the embodiment. [Figure 17] Figure 17 is a flowchart showing another determination method according to the embodiment. [Figure 18] Figure 18 is a schematic diagram showing another example of output from the determination system according to the embodiment. [Figure 19] Figure 19 is a schematic diagram showing another example of a cross-reality device according to the embodiment. [Figure 20] Figure 20 is a schematic diagram representing the hardware configuration. [Modes for carrying out the invention]

[0007] The embodiments of the present invention will be described below with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the ratio of the sizes of the parts, etc., are not necessarily the same as those of reality. Furthermore, even when representing the same part, the dimensions and ratios may be represented differently in the drawings. In this specification and each drawing, elements similar to those already described are denoted by the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] When a person moves their body, forces act on the muscles, joints, and tendons of the body. If the force acting on the body is large, it may cause pain to the muscles or joints or lead to serious injuries. For example, the force applied to a joint is related to the forces acting on various parts of the body when a person moves. The acting force in the body is represented by muscle activity amount, generated muscle strength, antagonist muscle strength, joint force, joint moment, and so on. As an example, the greater the muscle activity amount, the greater the force exerted by the muscle, and the load corresponding to the muscle strength is applied to the joint. Therefore, in order to examine the load on a specific part of the body, it is effective to examine the acting force in the body.

[0009] In addition, the acting force in the body also depends on how the person moves their body. For example, even when performing the same movement, the acting force generated during the movement of a skilled person is different from that generated during the movement of a beginner. If the movement of a beginner can be improved, it will lead to increased movement efficiency and prevention of body injuries.

[0010] One aspect of the embodiments of the present invention aims to provide a technique that can easily determine the load, risk, or proficiency level related to work. Another aspect of the embodiments of the present invention aims to train a model so that the acting force in the body can be more easily estimated.

[0011] FIG. 1 is a flowchart showing a processing method according to an embodiment. The processing method according to the embodiment mainly includes learning (step S10) and determination (step S20). In learning (step S10), an estimation model for estimating the acting force in the body is trained using various data (step S11). Further, a risk determination model for determining the risk in work may be trained (step S12). A proficiency determination model for determining the proficiency level of a worker may be trained (step S13). The order of steps S11 to S13 can be changed.

[0012] In the determination (step S20), the acting force when a person is actually moving their body is estimated using a learned estimation model (step S21). Further, a load determination (step S22) is executed using the estimation result of the acting force. Additionally, a risk determination (step S23) and a proficiency determination (step S24) may be executed. In the risk determination, a learned risk determination model is used. In the proficiency determination, a learned proficiency determination model is used. The order of steps S22 to S24 can be changed.

[0013] Hereinafter, a specific example of each process will be described. Here, the case where a person performs a screwing operation using a digital torque wrench will be described.

[0014] FIG. 2 is a schematic diagram showing the configuration of the determination system according to the embodiment. As shown in FIG. 2, the determination system 1 according to the embodiment includes a digital torque wrench 10, a photographing device 20, and a processing device 30. In the illustrated example, the operator W is using the digital torque wrench 10 to screw the member M. The digital torque wrench 10 can detect the torque when the screw is rotated. The digital torque wrench 10 is an example of a device including a sensor. The photographing device 20 photographs the operator W during the operation.

[0015] The processing device 30 receives an image of the operator W from the photographing device 20. The processing device 30 performs pose estimation on the operator W in the image. By the pose estimation, the positions of each joint of the operator W are estimated. A pose estimation model such as Dark Pose or Open Pose can be used for the pose estimation. For example, by the pose estimation, the positions of each joint such as the head, neck, shoulders, elbows, hands, fingers, waist, knees, and feet are estimated. The processing device 30 calculates the coordinates of each joint from the estimation result. The processing device 30 calculates the angle of each joint by inverse kinematics calculation using the coordinates of each joint.

[0016] The processing unit 30 receives the detected torque from the digital torque wrench 10. The magnitude of the torque is related to the force applied by the worker W to the digital torque wrench 10. In other words, the detected torque corresponds to the external force generated by the worker W's movement. The processing unit 30 uses the calculated coordinates of each joint and the detected torque to calculate the moment (torque) of each joint by inverse dynamics calculation. The processing unit 30 also calculates the acting forces such as muscle activity, generated muscle force, antagonistic muscle force, and joint force from the calculated joint moments. Existing musculoskeletal simulations can be used to calculate muscle activity, generated muscle force, antagonistic muscle force, joint force, and joint moment.

[0017] In addition to the applied force, the coordinates of the worker W's hand are acquired. The hand coordinates may be extracted from the results of pose estimation. Alternatively, in the example shown in Figure 2, worker W is working while wearing a mixed reality (MR) device 100. The MR device 100 can display various information to worker W. The MR device 100 also includes a camera and is capable of hand tracking. In hand tracking, worker W's hand is detected from the image and the coordinates of worker W's hand are calculated. The hand coordinates calculated by the MR device 100 may be referenced.

[0018] The digital torque wrench 10 continuously detects torque. The imaging device 20 repeatedly takes images. The processing device 30 repeatedly calculates joint coordinates and applied forces. The MR device 100 repeatedly performs hand tracking. This yields time-series data of hand coordinates, torque, and applied forces. From this time-series data, the processing device 30 generates datasets of hand coordinates, torque, and applied forces at arbitrary timings. Datasets of hand coordinates, torque, and applied forces are prepared for multiple timings.

[0019] The following sections will primarily explain the case where the force acting is muscle activity.

[0020] Figures 3 and 4 are tables illustrating the dataset. Table 40, shown in Figure 3, includes the hand coordinates 41 and torque 42 per second. Table 50, shown in Figure 4, shows the muscle activity in each part of the body per second. In other words, in the illustrated example, a dataset is provided for each second.

[0021] Figure 5 is a schematic diagram illustrating the learning method of the estimation model. As shown in Figure 5, the processing unit 30 trains an estimation model M1 for estimating the force acting on each part of the body. At this time, the coordinates and torque of the hand are used as input data, and the force acting is used as output data. The estimation model M1 is supervised learning using multiple prepared datasets. As a result, the estimation model M1 is learned to output (estimate) the force acting in response to the input of the hand coordinates and torque. To improve estimation accuracy, it is preferable that the estimation model M1 includes a neural network. It is more preferable that the estimation model M1 includes a graph neural network (GNN) or a recurrent neural network (RNN). The RNN preferably includes a Long Short-Term Memory (LSTM) structure.

[0022] Figure 6 is a flowchart showing the learning method of the estimation model. When worker W starts working, the MR device 100 performs hand tracking and obtains the coordinates of the hand (step S11a). The digital torque wrench 10 detects the torque (step S11b). The imaging device 20 takes a picture of worker W while he is working (step S11c). The processing device 30 uses the image acquired by the imaging device 20 to estimate his posture (step S11d). Based on the posture estimation, the processing device 30 calculates the coordinates of each joint of worker W (step S11e). The processing device 30 calculates the forces acting on each part of the body by inverse kinematics calculation and inverse dynamics calculation (step S11f).

[0023] Steps S11a, S11b, and S11f prepare multiple datasets at multiple time points. The processing unit 30 trains the estimation model M1 using the multiple datasets (step S11g). The processing unit 30 saves the trained estimation model M1 (step S11h).

[0024] In the example described above, the estimation model M1 is trained using short-term forces measured every second. Instead of this training, or in addition to it, training using long-term forces may be performed. For example, longer-term muscle activity can be calculated by averaging the muscle activity obtained over a period of 2 to 5 seconds. In addition, the hand coordinates and torque at some point during that period are obtained. Another estimation model is trained using the dataset of hand coordinates, torque, and long-term muscle activity.

[0025] Short-term forces can lead to sudden strains and potentially cause conditions like lumbago (acute lower back pain). Long-term forces can lead to fatigue and potentially cause conditions like tenosynovitis, runner's knee (patellofemoral pain syndrome), and tennis elbow (lateral epicondylitis).

[0026] In addition to the training of the muscle activity estimation model described above, the training of the risk assessment model and the proficiency assessment model are also performed.

[0027] Figure 7 is a flowchart illustrating the learning method of the risk assessment model. Figure 8 is a schematic diagram showing the learning method of the risk assessment model. First, prepare a dataset to be used to train the risk assessment model (step S12a). The dataset includes forces and labels for those forces. The labels indicate whether or not there is a risk associated with the force. The dataset may use forces obtained in step S11f, or forces output from a trained estimation model.

[0028] For example, if muscle activity in a particular area is high and there is a possibility of damaging a joint or muscle, a label indicating a risk will be attached. If muscle activity is low and there is no possibility of damaging a joint or muscle, a label indicating no risk will be attached.

[0029] Multiple labels may be provided to indicate the type of risk. For example, multiple labels may be provided to indicate specific risks such as lumbago (acute lower back pain), tenosynovitis, runner's knee, and tennis elbow.

[0030] Labels and the corresponding forces can be prepared using musculoskeletal simulation. For example, the processing unit 30 simulates movements that occur when there is a risk of lower back pain. The processing unit 30 acquires the forces acting on each part of the body at that time. This provides a dataset of forces acting on each part and their labels (lower back pain). Similarly, datasets for other types of risks can also be prepared using musculoskeletal simulation.

[0031] As shown in Figure 8, the processing unit 30 uses the applied force 60 as input data and the label 61 as output data to supervise training of the hazard determination model M2 (step S12b in Figure 7). As a result, the hazard determination model is trained to output a determination result of whether or not there is a hazard, in response to the input of the applied force. To improve the accuracy of the determination, it is preferable that the hazard determination model includes a neural network. The processing unit 30 then stores the trained hazard determination model M2 (step S12c).

[0032] Figure 9 is a flowchart illustrating the learning method of the proficiency assessment model. Figure 10 is a schematic diagram showing the learning method of the proficiency assessment model. First, a dataset is prepared to be used to train the proficiency assessment model (step S13a). The dataset includes the forces acting on each part of the body and labels for those forces. The labels indicate the level of proficiency in the task for each force. For example, proficiency is expressed numerically, with higher numbers indicating better performance.

[0033] As an example, skill level is evaluated based on the time required for the task. Those who complete the task relatively quickly are evaluated as "skilled," while those who complete it relatively quickly are evaluated as "beginners." The force exerted by each worker during the task is recorded. This results in a label (skilled) and the force exerted corresponding to that label.

[0034] As shown in Figure 10, the processing unit 30 uses the applied force 65 as input data and the label 66 as output data to supervise the training of the proficiency determination model M3 (step S13b in Figure 9). As a result, the proficiency determination model is trained to output a determination result of the worker's proficiency level in response to the applied force input at each part. To improve the accuracy of the determination, the proficiency determination model preferably includes a neural network. The processing unit 30 then stores the trained proficiency determination model M3 (step S13c).

[0035] Through the above process, a force estimation model, a risk assessment model, and a skill level assessment model are prepared.

[0036] Figure 11 is a flowchart showing the determination method according to the embodiment. When worker W starts working, the coordinates of the hand are acquired (step S21a) and the torque is detected (step S21b). The hand coordinates may be calculated using images from the imaging device 20, or acquired using the results of hand tracking from the MR device 100. The calculation of hand coordinates and the detection of torque are repeated while worker W is working.

[0037] Preferably, the results of hand tracking from the MR device 100 are used. When the worker wears the MR device 100, the MR device 100 is positioned close to the worker W's hands. From the perspective of the MR device 100, the hands are less likely to be obscured by objects, and the hands are easier to detect compared to the imaging device 20. By using the results of hand tracking, the accuracy of the hand coordinates can be improved.

[0038] The processing unit 30 inputs the hand coordinates and torque at any given time into a trained estimation model. The estimation model M1 outputs the force acting in response to the input of hand coordinates and torque. The processing unit 30 acquires the force acting output from the estimation model M1.

[0039] As a concrete example, a first estimation model is prepared to estimate short-term effects, and a second estimation model is prepared to estimate long-term effects. The first estimation model is trained using a dataset collected every second, as explained with reference to Figure 3. The second estimation model is trained using a dataset collected every 2 to 5 seconds.

[0040] The processing unit 30 inputs the hand coordinates and torque into the first estimation model (step S21c). The processing unit 30 obtains the first estimation result output from the first estimation model (step S21d). The processing unit 30 also inputs the hand coordinates and torque into the second estimation model (step S21e). The processing unit 30 obtains the second estimation result output from the second estimation model (step S21f).

[0041] The processing unit 30 compares the first estimation result output from the first estimation model with a preset first threshold (step S22a). For example, one or more thresholds are preset for the first estimation result. The processing unit 30 determines the short-term load on the worker W from the comparison result between the first applied force and the threshold (step S22b).

[0042] The processing unit 30 compares the second estimation result output from the second estimation model with a preset second threshold (step S22c). For example, one or more thresholds are preset for the second estimation result. The processing unit 30 determines the long-term load on worker W from the comparison result between the second estimation result and the threshold (step S22d).

[0043] Next, the processing unit 30 inputs the force applied in step S21d into the hazard assessment model (step S23a). The processing unit 30 then obtains the hazard assessment result output from the hazard assessment model (step S23b). The hazard assessment model is used to detect hazards that are difficult to determine by simply comparing the force applied with a threshold. By using the hazard assessment model to determine the presence or absence of hazards, the risk of injury to workers can be determined more accurately.

[0044] The processing unit 30 inputs the force acquired in step S21d into the proficiency determination model (step S24a). The processing unit 30 acquires the proficiency determination result output from the proficiency determination model (step S24b). The processing unit 30 outputs the obtained result (step S25).

[0045] Figure 12 is a table illustrating the output results from the first estimation model. For example, during operation, the hand coordinates and torque are acquired every second, as shown in Figure 3. The processing unit 30 inputs the hand coordinates and torque into the first estimation model to obtain the estimation result 70 shown in Figure 12. In the example shown in Figure 12, joint moment, muscle activity, and generated muscle force are estimated every second for each part of the body.

[0046] Figure 13 is a table illustrating thresholds for determining the load. The estimation results shown in Figure 12 are compared with the thresholds shown in Figure 13. In Table 80 shown in Figure 13, multiple thresholds are set in the muscle activity column 81 and the generated muscle force column 82. The muscle activity and generated muscle force for each body part included in the estimation results are compared with the thresholds defined in columns 81 and 82, respectively. Based on the comparison results, the load level defined in column 83 is determined. Using the data shown in Figures 12 and 13, the load level for each part of the body is determined every second.

[0047] If a second estimation model is used, estimation results are also obtained from the second estimation model. A threshold is set for the estimation results from the second estimation model. The processing unit 30 determines the long-term load level on each part of the body by comparing the estimation results obtained from the second estimation model with the threshold.

[0048] The processing unit 30 further inputs the force output from the estimation model into the hazard determination model and the proficiency determination model to obtain the determination results for the presence or absence of hazard and the determination results for the proficiency of the worker.

[0049] If the assessment results indicate a load or danger, it is desirable that this result be communicated to the worker. For example, an alert based on the assessment results may be displayed on the MR device 100 or output as an audio message from the MR device 100.

[0050] The presence or absence of a load may be determined based on the applied force, or the load level may be classified into three or more categories, as shown in Figure 13. The content of the output may change according to the classification result of the load level. Similarly, the presence or absence of a hazard may be determined based on the applied force, or the hazard level may be classified into three or more categories. The content of the output may change according to the classification result of the hazard level.

[0051] Figure 14 is a flowchart illustrating output control based on the load determination result. For example, in step S25 shown in Figure 11, the processing unit 30 may execute the process shown in Figure 14. The processing unit 30 refers to the load determination result (step S25a). If the load level is "1", the processing unit 30 does not output an alert (step S25b). A load level of 1 means that there is virtually no load. If the load level is "2", the processing unit 30 outputs a cautionary alert (step S25c). If the load level is "3", the processing unit 30 outputs a warning alert (step S25d). For example, in step S25d, the alert is displayed with a stronger warning color than in step S25c. In step S25d, a larger alert may be displayed than in step S25c. Also, in step S25d, the alert may be output with a louder voice than in step S25c.

[0052] Figure 15 is a schematic diagram showing an example of a cross-reality device according to an embodiment. Figure 15 shows an MR device as an example of a cross-reality device. As shown in Figure 15, the MR device 100 includes a frame 101, lenses 111 and 112, projection devices 121 and 122, an image camera 131, a depth camera 132, a sensor 140, a microphone 141, a processing unit 150, a battery 160, and a storage device 170.

[0053] In the illustrated example, the MR device 100 is a two-lens head-mounted display. Two lenses 111 and 112 are fitted into the frame 101. Projection devices 121 and 122 project information onto lenses 111 and 112, respectively.

[0054] Projection devices 121 and 122 display the results of recognizing the worker's body, virtual objects, etc., on lenses 111 and 112. Alternatively, only one of the projection devices 121 or 122 may be provided, and information may be displayed on only one of the lenses 111 or 112.

[0055] Lenses 111 and 112 are light-transmitting. The worker can see the real world through lenses 111 and 112. The worker can also see the information projected onto lenses 111 and 112 by projection devices 121 and 122. The information is superimposed onto the real space by projection by projection devices 121 and 122.

[0056] Image camera 131 detects visible light and obtains a two-dimensional image. Depth camera 132 emits infrared light and obtains a depth image based on the reflected infrared light. Sensor 140 is a 6-axis detection sensor capable of detecting angular velocity in 3 axes and acceleration in 3 axes. Microphone 141 accepts voice input.

[0057] The processing unit 150 controls each element of the MR device 100. For example, the processing unit 150 controls the display by the projection device 121 and projection device 122. The processing unit 150 detects movement of the field of view based on the detection results from the sensor 140. The processing unit 150 changes the display by the projection device 121 and projection device 122 in accordance with the movement of the field of view. In addition, the processing unit 150 can perform various processes using data obtained from the image camera 131 and depth camera 132, data from the storage device 170, etc.

[0058] The battery 160 supplies the power necessary for operation to each element of the MR device 100. The storage device 170 stores data necessary for processing by the processing unit 150, data obtained from processing by the processing unit 150, etc. The storage device 170 may be located outside the MR device 100 and may communicate with the processing unit 150.

[0059] The MR device according to this embodiment may be a single-lens head-mounted display, not limited to the illustrated example. The MR device may be a glasses-type device as shown in the illustration, or a helmet-type device.

[0060] The processing unit 150 may also have the functionality of the processing unit 30. In that case, the processing unit 30 may be omitted in the determination system 1 shown in Figure 2. Alternatively, in the method described above, a part of the processing performed by the processing unit 30 may be performed by the processing unit 150.

[0061] Figure 16 is a schematic diagram showing an example of output from the determination system according to the embodiment. In the example shown in Figure 16, worker W is tightening a screw onto member M using a digital torque wrench 10 and extension bar 11. During the operation, the processing unit 150 performs hand tracking using the images captured by the image camera 131 and depth camera 132. Hand tracking recognizes worker W's hand and calculates the coordinates of the hand.

[0062] The processing unit 30 receives the detected torque value from the digital torque wrench 10. The processing unit 30 inputs the hand coordinates and the detected torque value into the estimation model and obtains the estimated force result. The processing unit 30 uses the estimation result to determine the load. The processing unit 30 also uses a hazard determination model and a proficiency determination model to determine the hazard and proficiency levels.

[0063] In the example shown in Figure 16, the force estimation results indicate that the worker's waist is under strain and is at risk. The processing unit 30 transmits these determination results to the MR device 100. The processing unit 150 displays an alert AL indicating that the waist is under strain and is at risk.

[0064] Figure 17 is a flowchart showing another determination method according to the embodiment. The determination results obtained by the determination method may be saved in association with the work data. For example, as shown in Figure 17, first the processing unit 30 identifies the work to be performed (step S20a). For example, an operator inputs the work to be performed to the processing unit 30. The processing unit 30 identifies the work to be performed by accepting this input. The processing unit 30 may also identify the work to be performed based on a pre-registered schedule and time. The processing unit 30 may also identify the work to be performed based on an image captured by the imaging device 20 or the image camera 131. For example, a component in the image is compared with a pre-prepared template image. Each template image is associated with one of the tasks. The processing unit 30 determines the template image that is most similar to the component in the image by template matching. The processing unit 30 identifies the task associated with that template image as the work to be performed.

[0065] The processing unit 30 performs force estimation (step S21), load determination (step S22), hazard determination (step S23), and skill level determination (step S24). The processing unit 30 outputs each determination result (step S25). The processing unit 30 also links each determination result to the work data identified in step S20a and saves it as historical data (step S26).

[0066] The processing unit 30 determines whether all operations have been completed (step S27). If not all operations have been completed, step S20a is executed again. As a result, the multiple determination results obtained in step S20 are linked to the data of each operation.

[0067] Figure 18 is a schematic diagram showing another example of output from the determination system according to the embodiment. In the example shown in Figure 18, the skill level of each worker for each process is displayed in a radar chart 85. For example, workers W1 and W2 perform processes A through E. This allows for the determination of the skill levels of workers W1 and W2 for each of processes A through E. In the illustrated example, skill levels are evaluated on a 6-point scale from "0" to "5". A higher number indicates a higher skill level.

[0068] As shown in Figure 18, the proficiency level for each task is displayed for each worker, making it easy to understand each worker's strengths and weaknesses. For example, for tasks where proficiency is judged to be low, skilled workers can instruct beginners to improve the beginners' work efficiency. For tasks where proficiency is judged to be high, sharing videos of that task among multiple workers can improve each worker's proficiency.

[0069] Figure 19 is a schematic diagram showing another example of a cross-reality device according to the embodiment. Figure 19 shows a VR device as an example of a cross-reality device. The VR device 200 shown in Figure 19 is a goggle type. The VR device 200 comprises a monitor 210, a remote control 220, an image camera 231, a depth camera 232, and a processing unit 250. The monitor 210 displays a virtual space. The wearer views the virtual space displayed on the monitor 210.

[0070] The wearer holds and operates the remote control 220. The remote control 220 includes one or more sensors selected from the group consisting of an accelerometer and an angular velocity sensor. The remote control 220 is an example of a device that includes sensors. When the remote control 220 is operated, a signal is transmitted from the remote control 220 to the processing unit 250. For example, the wearer of the VR device 200 can play a video game on the VR device 200 while operating the remote control 220.

[0071] Image camera 231 and depth camera 232 capture images in front of the wearer. Processing unit 250 uses the captured images to perform hand tracking. This calculates the coordinates of the wearer's hands.

[0072] The processing unit 30 (or processing unit 250) uses the hand coordinates and the values ​​detected by the remote control 220 (acceleration or angular velocity) to perform the determination shown in Figure 1 (step S20). This determines the wearer's load, risk, and skill level.

[0073] The advantages of one aspect of the embodiment will be explained. When a person moves, forces act on their body. For example, muscles generate force, which in turn applies force to the joints. These forces acting on the body can unintentionally or unconsciously strain the body, potentially causing pain or injury. In one embodiment of the present invention, the forces acting on the body are used to determine the load or risk to a person in order to prevent physical injury. By determining the load or risk, and if a load or risk is determined to exist, the movement can be stopped or modified to avoid physical injury. Furthermore, the forces acting on the body are obtained by inputting the coordinates of the hand and the values ​​detected by the device's sensors into an estimation model. Therefore, there is no need to prepare various sensors to estimate the forces acting on the body. For example, the data necessary for estimating the forces acting on the body can be obtained without hindering human movement.

[0074] The advantages of another aspect of the embodiment will be described. When a person performs a specific action, accuracy or efficiency of that action is sometimes required. For example, accurate or quick movements during work can improve work efficiency. When playing a game, accurate movements allow for better gameplay. In one embodiment of the present invention, applied force is used to determine a person's skill level. As described above, applied force is estimated using the coordinates of the hand and values ​​detected by sensors. Therefore, according to this embodiment, skill level can be determined more easily.

[0075] According to embodiments of the present invention, the burden on a person, the risk to a person, or the skill level of a person can be determined more easily. Furthermore, according to embodiments, the data necessary for determining the burden, risk, or skill level can be acquired without interfering with the person's actions.

[0076] The embodiment is particularly suitable for specific tasks that are performed repeatedly. For example, when a product is manufactured, a large number of screws are tightened. For instance, in the production of large products such as generators and plants, thousands of screws are tightened in each product. The person performing the product assembly repeatedly tightens screws, which places a very heavy burden on the body. Furthermore, there are various regulations regarding the order of screw tightening, the tightening strength, etc., and the work must be performed in accordance with these regulations. As a result, the progress of the work varies greatly depending on the skill level of the worker. For example, in order to proceed with the work according to the work schedule, it is necessary to more accurately understand the skill level of each worker.

[0077] According to the embodiment, physical strain or danger can be easily determined. If strain or danger is determined, an alert AL is output as shown in Figure 16. By encouraging workers to improve their posture, physical injuries can be avoided. Alternatively, according to the embodiment, skill level can be easily determined. By digitizing skill level, people can be assigned to each task more appropriately, and the accuracy of work schedule progress can be improved. Furthermore, by training workers based on the skill level determination results, personnel training can be carried out more efficiently.

[0078] Furthermore, according to the embodiment, a model including a neural network is trained using hand coordinates, sensor-detected values, and applied force. Hand coordinates can be easily obtained, for example, using a camera in a cross-reality device. Detected values ​​can be easily obtained by a device including sensors. Acquiring this data does not easily interfere with human movement. According to the trained model, the applied force can be estimated using this data. For example, the estimated applied force can be used to determine the load on the person, the risk to the person, or the person's skill level, as described above.

[0079] In the embodiments described above, an example was explained in which three factors—load on a person, risk to a person, and skill level of a person—are determined using applied force. The embodiments are not limited to this example. For example, only one or two selected from the group consisting of load on a person, risk to a person, and skill level of a person may be determined.

[0080] Figure 20 is a schematic diagram representing the hardware configuration. The processing unit 30, 150, or 250 includes, for example, the computer 90 shown in Figure 20. The computer 90 includes a processing circuit 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0081] ROM92 stores programs that control the operation of computer 90. ROM92 contains the programs necessary for computer 90 to perform each of the processes described above. RAM93 functions as a memory area where the programs stored in ROM92 are loaded.

[0082] The processing circuit 91 includes an arithmetic processing unit such as a CPU or GPU. The processing circuit 91 uses RAM 93 as work memory and executes a program stored in at least one of ROM 92 or storage device 94. During program execution, the processing circuit 91 controls each component via the system bus 98 and performs various processes.

[0083] The memory device 94 stores data necessary for program execution and data obtained through program execution.

[0084] The input interface (I / F) 95 can connect the computer 90 and the input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The processing circuit 91 can read various data from the input device 95a via the input I / F 95.

[0085] The output interface (I / F) 96 can connect the computer 90 and the output device 96a. The output I / F 96 is a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI®). The processing circuit 91 can transmit data to the output device 96a via the output I / F 96 and display an image on the output device 96a.

[0086] The communication interface (I / F) 97 can connect the computer 90 to a server 97a located outside the computer 90. The communication I / F 97 is, for example, a network card such as a LAN card. The processing circuit 91 can read various data from the server 97a via the communication I / F 97.

[0087] The storage device 94 includes one or more selected from Hard Disk Drives (HDDs) and Solid State Drives (SSDs). The input device 95a includes one or more selected from a mouse, keyboard, microphone (voice input), and touchpad. The output device 96a includes one or more selected from a monitor, projector, printer, and speaker. Devices that have the functions of both input device 95a and output device 96a, such as a touch panel, may also be used.

[0088] The processing required for the determination method may be performed by a single computer 90, or it may be performed collaboratively by multiple computers 90.

[0089] The processing of the various data described above may be recorded as a program that can be executed by a computer on a magnetic disk (flexible disk and hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, etc.), a semiconductor memory, or another non-transitory computer-readable storage medium.

[0090] For example, data on a recording medium is read by a computer (processing circuit). The recording format (storage format) on the recording medium is arbitrary. For example, a computer reads a program from the recording medium and causes the CPU to execute instructions based on this program. The acquisition (or reading) of the program by the computer may be performed via a network.

[0091] Embodiments of the present invention may be implemented as a determination device that causes a processing device to execute the program of the determination method described above. The determination device can determine the burden on a person, the danger to a person, or the skill level of a person. Furthermore, embodiments of the present invention may be implemented as a learning device that causes a processing device to execute the program of the learning method described above. The learning device uses coordinates and detected values ​​as input data and the applied force as output data to train a model including a neural network.

[0092] Embodiments of the present invention include the following features. (Feature 1) In the processing unit, From an image showing a human hand, the coordinates of the hand are measured. The sensor of the device held by the person receives the detected value, The coordinates and detected values ​​are input into an estimation model for estimating the force acting on the body. Using the force acting on the person's body output from the estimation model, one or more of the following are selected from the group consisting of the load on the person, the risk to the person, and the person's skill level. Judgment method. (Feature 2) The aforementioned processing apparatus, By comparing the aforementioned applied force with a preset threshold, the load is determined. If a load is detected, an alert will be issued. The determination method described in Feature 1. (Feature 3) The aforementioned processing apparatus, By inputting the aforementioned force into a risk assessment model for determining risk, the risk is determined. An alert will be issued if a risk is detected. The determination method described in Feature 1 or 2. (Feature 4) The aforementioned processing apparatus, By inputting the aforementioned force into a proficiency determination model for determining proficiency, the proficiency level of the person is determined. Output the determined proficiency level. The determination method described in one of the features 1 to 3. (Feature 5) The determination method described in any one of features 1 to 4, wherein the aforementioned force is one or more selected from the group consisting of muscle activity, generated muscle force, antagonistic muscle force, joint force, and joint moment. (Feature 6) Equipped with a processing device, A determination device that causes the processing device to execute the determination method described in any one of features 1 to 5. (Feature 7) The determination device described in Feature 6, A camera that photographs a person's hand, The aforementioned device and, A judgment system equipped with the following features. (Feature 8) The determination system according to feature 7, wherein the sensor detects one or more selected from the group consisting of torque, acceleration, and angular velocity. (Feature 9) Processing circuit and It includes a camera that photographs a person's hand, A cross-reality device that causes a processing circuit to execute one of the determination methods described in one of the features 1 to 5. (Feature 10) A program that causes the processing unit to execute the determination method described in Feature 1. (Feature 11) In the processing unit, The device acquires the coordinates of the hand of the person holding the device, the values ​​detected by the device's sensors, and the force acting on the person's body. The aforementioned coordinates and the detected values ​​are used as input data, and the aforementioned force is used as output data to train a model including a neural network. Learning methods. (Feature 12) Equipped with a processing circuit, A learning device that causes a processing circuit to execute the learning method described in Feature 11. (Feature 13) A program that causes the processing unit to execute the learning method described in Feature 11. (Feature 14) A storage medium storing the program described in feature 10 or 13.

[0093] The embodiments described above provide a determination method, determination device, determination system, cross-reality device, program, and storage medium that can more easily determine the burden on a person, the danger to a person, or the skill level of a person. Furthermore, a learning method, learning device, program, and storage medium are provided that can learn a model so that it can estimate the force acting on a person's body using easily obtainable data.

[0094] In this specification, "or" indicates that "at least one" of the items listed in the text may be adopted.

[0095] Although several embodiments of the present invention have been illustrated above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. Furthermore, the embodiments described above can be implemented in combination with each other. [Explanation of Symbols]

[0096] 1: Judgment system, 10: Digital torque wrench, 11: Extension bar, 20: Imaging device, 30: Processing device, 90: Computer, 100: MR device, 101: Frame, 111,112: Lens, 121,122: Projection device, 131: Image camera, 132: Depth camera, 140: Sensor, 141: Microphone, 150: Processing device, 160: Battery, 170: Memory device, 200: VR device, 210: Monitor, 220: Remote control, 231: Image camera, 232: Depth camera, 250: Processing device, M: Material, M1: Estimation model, M2: Hazard assessment model, M3: Skill assessment model, W: Worker

Claims

1. In the processing unit, From an image showing a human hand, the coordinates of the hand are measured. The sensor of the device held by the person receives the detected value, The coordinates and detected values ​​are input into an estimation model for estimating the force acting on the body. Using the force acting on the person's body output from the estimation model, one or more of the following are selected from the group consisting of the load on the person, the risk to the person, and the person's skill level. Judgment method.

2. The aforementioned processing apparatus, By comparing the aforementioned applied force with a preset threshold, the load is determined. If a load is detected, an alert will be issued. The determination method according to claim 1.

3. The aforementioned processing apparatus, By inputting the aforementioned force into a risk assessment model for determining risk, the risk is determined. An alert will be issued if a risk is detected. The determination method according to claim 1.

4. The aforementioned processing apparatus, By inputting the aforementioned force into a proficiency determination model for determining proficiency, the proficiency level of the person is determined. Output the determined proficiency level. The determination method according to claim 1.

5. The determination method according to claim 1, wherein the applied force is one or more selected from the group consisting of muscle activity, generated muscle force, antagonistic muscle force, joint force, and joint moment.

6. Equipped with a processing device, A determination device that causes the processing device to execute the determination method described in any one of claims 1 to 5.

7. The determination device according to claim 6, A camera that photographs a person's hand, The aforementioned device and, A judgment system equipped with the following features.

8. The determination system according to claim 7, wherein the sensor detects one or more selected from the group consisting of torque, acceleration, and angular velocity.

9. Processing circuit and It includes a camera that photographs a person's hand, A cross-reality device that causes the processing circuit to execute the determination method described in any one of claims 1 to 5.

10. A program that causes a processing device to execute the determination method described in claim 1.

11. In the processing unit, The device acquires the coordinates of the hand of the person holding the device, the values ​​detected by the device's sensors, and the force acting on the person's body. The aforementioned coordinates and detected values ​​are used as input data, and the aforementioned force is used as output data to train a model including a neural network. Learning methods.

12. Equipped with a processing circuit, A learning device that causes a processing circuit to execute the learning method described in claim 11.

13. A program that causes a processing device to execute the learning method described in claim 11.

14. A storage medium storing the program described in claim 10 or 13.

Images