Information processing device, information processing method, and information processing program

Abstract

Workers can learn the job with peace of mind. [Solution] An information processing device 10 includes: a reception unit that receives a video of the worker's movements captured by a camera 50; a generation unit that recognizes the joints of the worker's body in the video and generates time-series coordinate data, which is the time-series coordinate of each joint corresponding to the movements; an evaluation unit that evaluates the worker's movements using a score compiled based on the difference between the time-series coordinate data relating to the worker's movements and the time-series coordinate data relating to the movements of a skilled worker who serves as a model among the workers 20; and an output unit that outputs advice for the worker's movements corresponding to the score along with the score.

Description

Technical Field

[0004] ,

[0006] , , , ,

[0005] , , , , ,

[0001] The disclosed technology relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, in a work site such as a factory, there is a technology for quantitatively evaluating the proficiency of a worker's work (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the technology described in Patent Document 1 has aspects corresponding to management technology or monitoring technology, and the worker cannot concentrate on the work due to the sense of tension of being monitored, and there are problems in terms of ease of acquisition and work safety. <000002​​​​​​​​A first aspect of the present disclosure is an information processing device comprising: a receiving unit that receives a video of a worker's movements captured by an imaging device; a generating unit that recognizes the joints of the worker's body in the video and generates time-series coordinate data which are the time-series coordinates of each joint corresponding to the movements; an evaluation unit that evaluates the worker using a score aggregated based on the difference between the time-series coordinate data relating to the worker's movements and the time-series coordinate data relating to the movements of a skilled worker who serves as a model; and an output unit that outputs advice for the worker's movements corresponding to the score along with the score.

[0007] According to the information processing device of the first embodiment, workers can work on learning their tasks with peace of mind.

[0008] A second aspect of the present disclosure is an information processing apparatus of the first aspect, wherein the evaluation unit calculates the degree of difference between the time-series coordinate data of the skilled worker and the time-series coordinate data of the worker based on the distance between the time-series coordinate data of the two, and evaluates with the score which is a value obtained by aggregating the degree of difference.

[0009] According to the information processing device of the second embodiment, evaluation can be performed with a highly accurate score.

[0010] A third aspect of this disclosure is an information processing apparatus according to the first aspect, wherein the output unit outputs a video in which a model video in which the contours of each object are represented in a single color over the video of the skilled worker's actions and a current video in which the contours of each object are represented in a single color different from the single color over the video of the worker's actions.

[0011] According to the information processing device of the third embodiment, a model video and a current video can be visually compared.

[0012] A fourth aspect of this disclosure is an information processing apparatus of the third aspect, wherein when the worker's movement enters a predetermined location, the imaging device starts imaging, and when the worker's movement enters another predetermined location, the output unit outputs at least one of the score, the advice, and the superimposed video.

[0013] According to the information processing device of the fourth embodiment, the entire process from the start of imaging to the end of evaluation can be automated.

[0014] A fifth aspect of this disclosure is an information processing apparatus according to the first aspect, wherein the receiving unit receives video footage of the actions of multiple workers simultaneously captured by an imaging device, the evaluation unit evaluates the score for each of the multiple workers, and the output unit outputs the result of a relative evaluation of the scores of the multiple workers, reflecting the result in the advice.

[0015] According to the information processing device of the fifth embodiment, multiple people can compete for work scores.

[0016] A sixth aspect of this disclosure is an information processing method, wherein a computer receives a video of a worker's movements captured by an imaging device, recognizes the joints of the worker's body in the video, generates time-series coordinate data which are the time-series coordinates of each joint corresponding to the movements, evaluates the worker's movements using a score aggregated based on the difference between the time-series coordinate data relating to the worker's movements and the time-series coordinate data relating to the movements of a skilled worker who serves as a model, and outputs advice for the worker's movements corresponding to the score along with the score.

[0017] According to the information processing method of the sixth embodiment, workers can work on learning their tasks with peace of mind.

[0018] A seventh aspect of this disclosure is an information processing program which causes a computer to perform the following processing: receiving a video of a worker's movements captured by an imaging device; recognizing the joints of the worker's body in the video; generating time-series coordinate data which are the time-series coordinates of each joint corresponding to the movements; evaluating the worker's movements with a score aggregated based on the difference between the time-series coordinate data relating to the worker's movements and the time-series coordinate data relating to the movements of a skilled worker who serves as a model; and outputting advice for the worker's movements corresponding to the score along with the score.

[0019] According to the information processing program of the seventh aspect, an operator can work on work acquisition with confidence.

Effect of the Invention

[0020] According to the disclosed technology, an operator can work on work acquisition with confidence.

Brief Description of the Drawings

[0021] [Figure 1] It is a diagram showing the overall configuration of the information processing apparatus 10 and peripheral devices according to the present embodiment. [Figure 2] It is a block diagram showing the hardware configuration of the information processing apparatus 10 according to the present embodiment. [Figure 3] It is a block diagram showing the functional configuration of the information processing apparatus 10 according to the present embodiment. [Figure 4] It is an explanatory diagram showing the start point of the video received by the reception unit of the information processing apparatus 10 according to the present embodiment. [Figure 5] It is an explanatory diagram showing the recognition of each joint and the like by the information processing apparatus 10 according to the present embodiment. [Figure 6] It is an explanatory diagram showing the recognition of each joint and the like by the information processing apparatus 10 according to the present embodiment. [[ID=�2]] [Figure 7] It is an explanatory diagram showing the recognition of each joint and the like by the information processing apparatus 10 according to the present embodiment. [Figure 8] It is an explanatory diagram showing the recognition of each joint and the like by the information processing apparatus 10 according to the present embodiment. [Figure 9] It is an explanatory diagram showing the recognition of each joint and the like by the information processing apparatus 10 according to the present embodiment. [Figure 10] It is an explanatory diagram showing the recognition of each joint and the like by the information processing apparatus 10 according to the present embodiment for each frame of the video. [Figure 11] It is an explanatory diagram showing the end point of the video received by the reception unit of the information processing apparatus 10 according to the present embodiment. [Figure 12] It is an explanatory diagram showing the comparison between the reference video and the current video by the information processing apparatus 10 according to the present embodiment. [Figure 13]This is an explanatory diagram showing a comparison between a model video and a current video using the information processing device 10 according to this embodiment. [Figure 14] This is an explanatory diagram showing a comparison between a model video and a current video using the information processing device 10 according to this embodiment. [Figure 15] This is an explanatory diagram showing an output screen image from the information processing device 10 according to this embodiment. [Figure 16] This is a flowchart showing the score evaluation process of the information processing device 10 according to this embodiment. [Modes for carrying out the invention]

[0022] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, identical or equivalent components and parts are given the same reference numerals. Furthermore, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0023] The information processing device 10 according to this embodiment is connected to the camera 50 to receive video footage of the worker 20's work captured by the camera 50. The information processing device 10 is also connected to the display device 40 to display the output results of the output unit 104, which will be described later (see Figure 1). In Figure 1, the information processing device 10 and the camera 50 are shown as separate devices, but this is not limited to this arrangement, and the camera 50 may be provided inside the information processing device 10. For example, the information processing device 10 may be a laptop computer, and the webcam built into or attached to the laptop computer may be the camera 50. Also, in Figure 1, the information processing device 10 and the display device 40 are shown as separate devices, but this is not limited to this arrangement, and for example, if the information processing device 10 is a laptop computer, the liquid crystal display portion of the laptop computer may be the display device 40.

[0024] As shown in Figure 1, camera 50 captures images of worker 20 performing processing or other tasks using material 30, and transmits the captured video from camera 50 to information processing device 10. The information processing device 10 recognizes the joints, eyes, mouth, nose, and ears of worker 20 in the video received from camera 50 during the worker's work. In Figure 1, the material before work is indicated by reference numeral 30, the material during work is indicated by reference numeral 33, and the finished product is indicated by reference numeral 35.

[0025] The information processing device 10 is composed of a computer. For example, the information processing device 10 may be an edge computer, a desktop PC, a notebook computer, or a server. The information processing device 10 may also complete the score evaluation process according to this embodiment (details will be described later) within the information processing device 10 without being connected to a network such as the Internet.

[0026] Camera 50 is a video camera that captures video. For example, camera 50 is fixed on a tripod or the like and positioned in a location where it can capture the movements of worker 20. Alternatively, for example, if camera 50 is a webcam built into the information processing device 10, the information processing device 10 itself is positioned in a location where it can capture the movements of worker 20. In the example in Figure 1, camera 50 is positioned so that it can capture the movements of worker 20's hands from the ceiling in front of worker 20.

[0027] The display device 40 is a device that displays the output results from the information processing device 10, and is, for example, a liquid crystal display. The display device 40 may also be equipped with a speaker or the like as an audio output means.

[0028] Figure 2 is a block diagram showing the hardware configuration of the information processing device 10 in this embodiment. As shown in Figure 2, the information processing device 10 is composed of a CPU (Central Processing Unit) 11A, a GPU (Graphics Processing Unit) 11B, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, storage 14, an input / output I / F (Interface) 15, and a communication I / F 16. Each component is connected to the others via a bus 17 so that they can communicate with each other.

[0029] The CPU 11A is a central processing unit that executes various programs and controls each component. Specifically, the CPU 11A reads programs from the ROM 12 and executes them using the RAM 13 as a working area. The CPU 11A controls each of the above components and performs various calculations according to the programs stored in the ROM 12.

[0030] The GPU 11B primarily handles the calculations necessary for rendering images. The GPU 11B processes massive calculations at high speed. In this embodiment, since it is necessary to perform video processing at high speed, the information processing device 10 is equipped with the GPU 11B. Note that the GPU 11B does not necessarily have to be installed inside the information processing device 10; for example, the GPU 11B may be located on a cloud server, and the information processing device 10 may access the cloud server and utilize the GPU 11B (not shown).

[0031] The storage device, comprised of ROM12, stores various programs, including the operating system, and various data. ROM12 also stores a processing program for executing the score evaluation process described later.

[0032] The memory configured by RAM13 temporarily stores programs and data as a working area.

[0033] Storage 14 consists of HDDs (Hard Disk Drives) and SSDs (Solid State Drives), and stores various types of data.

[0034] The input / output interface 15 is an interface for communicating with a display device 40, which is an example of an input and output device located outside the information processing device 10. The input device includes a camera 50, a pointing device such as a mouse, and a keyboard, and is used for various types of input. The output device is, for example, a liquid crystal display, and is a device for outputting various types of information. The output device may also function as an input device by employing a touch panel system. The output device may also be equipped with a speaker or the like as a means of audio output.

[0035] The communication interface 16 is an interface for communicating with other devices, such as a cloud server, located outside the information processing device 10. For this communication, a wired communication standard such as Ethernet® or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi® may be used. Note that if the information processing device 10 does not use network communication such as the Internet, it may complete the score evaluation process within the information processing device 10 without providing the communication interface 16.

[0036] Next, the functional configuration of the information processing device 10 according to this embodiment will be described with reference to Figure 3. As shown in Figure 3, the information processing device 10 of this embodiment includes a receiving unit 101, a generation unit 102, an evaluation unit 103, and an output unit 104. A data storage unit 105 is also provided in a predetermined storage area of ​​the ROM 12 or storage 14. The CPU 11A executes a processing program stored in the ROM 12 to function as the receiving unit 101, generation unit 102, evaluation unit 103, and output unit 104. Note that the processing of each of these functional units may be executed by the GPU 11B instead of the CPU 11A. The data storage unit 105 also stores evaluated scores, etc., which will be described later. The data storage unit 105 also stores captured video footage received by the receiving unit 101. Note that if the information processing device 10 has a communication I / F 16, the evaluated scores, etc., which will be described later, and the captured video footage received by the receiving unit 101 may be stored on a cloud server or the like located outside the information processing device 10.

[0037] The reception unit 101 receives video footage of the worker's movements captured by the camera 50, which is an imaging device. Specifically, the camera 50 transmits the video footage of the worker's movements to the information processing device 10, and the reception unit 101 of the information processing device 10 receives the video and accepts it as input.

[0038] The reception unit 101 initiates imaging by the camera 50 when the worker's movements enter a predetermined position. Figure 4 is an illustrative diagram of the video displayed on the display device 40. As an example, as shown in Figure 4, the starting position is set in advance as an area enclosed by a rectangular block 41, and for example, when both wrists 61 of the worker enter the rectangular block 41, the camera 50 starts imaging. The position of the rectangular block 41 can be flexibly set. The reception unit 101 receives the video when the camera 50 has started and stopped imaging.

[0039] Furthermore, the reception unit 101 also receives videos of the actions of multiple workers 20 simultaneously captured by the camera 50, which is an imaging device. Specifically, the reception unit 101 receives videos of multiple workers 20 working simultaneously, captured at the action level.

[0040] The video received by the reception unit 101 is stored in the data storage unit 105 of the information processing device 10, for example. When the output unit 104, described later, outputs a video that combines the video of the worker 20 and the video of the expert, the video stored in the data storage unit 105 is output, for example. In other words, the video output by the output unit 104 is a recorded video, and real-time video output is not basically performed. As a result, the worker 20 does not receive notifications such as alerts in real time while working, so the worker 20 can concentrate on the work and the safety of the work can be ensured. Technically, it is possible for the output unit 104 to output the video received by the reception unit 101 of the information processing device 10 in real time. However, from the perspective of improving work safety, in this embodiment, it is assumed that the information processing device 10 will not basically perform real-time video output.

[0041] The generation unit 102 recognizes the joints of the worker 20's body in the video and generates time-series coordinate data, which is the time-series coordinate of each joint according to its movement. Specifically, the generation unit 102 uses existing open-source skeletal detection technologies such as Openpose (https: / / github.com / CMU-Perceptual-Computing-Lab / openpose) and Posenet (https: / / www.tensorflow.org / lite / examples / pose_estimation / overview?hl=ja) to recognize the joints, eyes, mouth, nose, and ears (hereinafter referred to as "joints, etc.") of the worker 20's body in the video. Specifically, as shown in Figures 5 to 9, the generation unit 102 recognizes each joint, etc. of the worker 20's body in the video. To clarify, the generation unit 102 represents each joint, etc. as a point and connects each point with a line to show the overall movement.

[0042] As shown in Figure 10, the recognition of each joint, etc., by the generation unit 102 is performed for each frame of the video. In Figure 10, the recognition of each joint, etc., is performed on a frame-by-frame basis, and a portion of the total frames in the video are shown. The generation unit 102 also recognizes the position coordinates of each joint, etc., of the worker 20 shown in each frame, using XY coordinates.

[0043] The generation unit 102 plots the XY coordinates, which are the position coordinates of each recognized joint, etc., on a graph in chronological order and compares it with the graph of the skilled worker. When comparing the graph of the worker 20 with the graph of the skilled worker, the generation unit 102 uses a dynamic time warping method called DTW (Dynamic Time Warping). DTW is a method used to measure the similarity between time-series coordinate data and can determine the similarity of waveforms. Specifically, DTW calculates the distance to each point in the two time-series coordinate data by exhaustive calculation and extracts the path that minimizes the distance between each point in the two time-series coordinate data. Since overlap is allowed when selecting corresponding points, similarity can be calculated even if the length or period of the time-series coordinate data differs. In addition, with DTW, the closer the extracted shortest path is to a value of 0, the higher the similarity. In this embodiment, the distance between the time-series coordinate data is represented as the degree of difference, and the aggregated value of the degree of difference is used as the evaluation index score. For example, a score of 100 points occurs when the generation unit 102 compares the graph of worker 20 and the graph of expert using the DTW method, and the path with the shortest distance between each extracted point is valued at 0 (i.e., the difference is valued at 0), which means that the two time-series coordinate data are in perfect agreement.

[0044] The evaluation unit 103 evaluates the worker 20 using a score calculated based on the difference between the time-series coordinate data related to the worker 20's movements and the time-series coordinate data related to the movements of a skilled worker 20 who serves as a model. Here, the difference is the distance between the two sets of time-series coordinate data. Specifically, calculating this distance means that the generation unit 102 calculates the distance to each point by trial and error, extracts the path that minimizes the distance between each point in the two sets of time-series coordinate data, and the evaluation unit 103 calculates a score based on this path. The evaluation unit 103 evaluates the worker 20's movements using the calculated score. The specific formula for calculating the score is shown below. When a score of 100 points is given for a perfect match with the graph of a skilled worker representing standard work, the formula for calculating the score can be expressed as, for example, x = 1 ÷ (d ÷ 12 + 1) × 100. For example, if the value of the path d is 3, the score x will be 80 points. In the evaluation by the evaluation unit 103, there is a standard score, and the evaluation unit 103 sets the score at which the worker 20 is deemed to have mastered the task as the standard score. For example, if the standard score is set to 80 points, the evaluation unit 103 evaluates that a worker 20 who calculates a score of 80 points or higher has cleared the task, while the evaluation unit 103 evaluates a worker 20 who calculates a score of less than 80 points in a way that encourages them to try the task again. However, if the standard score is too strict, it may undermine the motivation of the worker 20 to work, so the standard score is set at a standard score.

[0045] The evaluation unit 103 calculates the degree of difference between the time-series coordinate data of the skilled worker and the time-series coordinate data of the worker 20 based on the distance between the two sets of time-series coordinate data, and evaluates them using a score that is an aggregated value of the degree of difference. Specifically, as an example, the evaluation unit 103 evaluates the actions of the worker 20 using a score calculated, for example, by the formula described above, based on the degree of difference in the time-series coordinate data, with a maximum score of 100 points.

[0046] Furthermore, the evaluation unit 103 evaluates each score for multiple people. Specifically, the evaluation unit 103 evaluates each score for multiple people simultaneously. Since the baseline score is determined by the work of an expert, the scores of multiple people are compared using absolute evaluation, ensuring that the baseline is the same and allowing for highly accurate relative evaluation.

[0047] The output unit 104 outputs advice for the worker's actions corresponding to the score, along with the score itself. Specifically, as shown in Figure 11, the output unit 104 outputs the score as a percentage of 100, and further outputs advice on areas for improvement and solutions. For example, as shown in Figure 11, the output unit 104 displays the score in the score display section 71 of the video on the display device 40, which is a composite of the current video and the example video. Alternatively, the output unit 104 displays text related to the advice in the advice display section 72 of the same composite video on the display device 40. Note that the score display content, advice content, layout, etc. in Figure 11 are examples only and are not limited thereto.

[0048] Furthermore, the output unit 104 outputs a video in which a model video, in which the outlines of each object are represented in a single color against a video of the skilled worker's actions, and a current video, in which the outlines of each object are represented in a single color different from the aforementioned single color against a video of the worker 20's actions, are superimposed. Here, if the video is of the skilled worker's actions, the object refers to at least the skilled worker and the objects related to the skilled worker's actions that are shown in the video. Similarly, if the video is of the worker's actions, the object refers to at least the worker and the objects related to the worker's actions that are shown in the video. Note that the single color can be any color that is easily visible. In addition to outputting the superimposed video superimposed on the model video of the skilled worker from the start to the end of the worker 20's work, it is also possible to extract and output points in the worker 20's current video that are particularly different from the model video, i.e., points where the actions are particularly out of sync. The worker 20 can not only check the sequence of actions in the superimposed video, but also check a summary that focuses on actions that particularly need improvement.

[0049] The output unit 104 outputs at least one of the score, advice, and overlaid video when the worker's 20's movements enter another predetermined location. For example, as shown in Figure 12, when the right hand 62 enters the rectangular block 42, the output unit outputs at least one of the score, advice, and overlaid video. The position of the rectangular block 42 can be flexibly set.

[0050] Specifically, as shown in Figures 13 to 15, the output unit 104 outputs, as an example, a score, advice, and an overlaid video. In Figures 13 to 15, the dotted lines represent the expert's exemplary movements, and the solid lines represent the worker's current movements. In the actual overlaid video, these are shown with different single colors to represent the outlines of each object. In Figure 13, the worker 20 is taking too long to place the material 30 on the table. In the example of Figure 13, as advice, the output unit 104 displays the text "Quickly remove the material 30" in the advice display section 72 to the display device 40. In Figure 14, the worker 20 is taking too long to finish the material 33 that he is working on. In the example of Figure 14, as advice, the output unit 104 displays the text "Reviewing and memorizing the finishing work will allow you to work more smoothly" in the advice display section 72 to the display device 40. Figure 15 shows that the actions of worker 20 are almost identical to those of a skilled worker. In the example in Figure 15, as advice, the output unit 104 displays the following text on the advice display section 72 of the display device 40: "That's good! You're getting the hang of it!" Although the example of advice output by the output unit 104 is shown as displaying advice on the advice display section 72 of the overlaid video, the output unit 104 is not limited to this and may also output advice via voice while the overlaid video is playing.

[0051] In this way, the output unit 104 displays and outputs a video that overlays the current video and the example video, allowing the worker 20 to visually review their own work. Furthermore, while outputting the overlaid video, the output unit 104 accurately indicates areas for improvement to the worker 20, encouraging them to learn. The worker 20 will develop a desire for a high score, and the advice and the presentation of challenges will stimulate their motivation to improve and continue working. In addition, since the output unit 104 outputs a score, it applies gamification to motivate the worker 20.

[0052] Furthermore, the output unit 104 outputs advice based on the results of a relative evaluation of the scores of multiple people (not shown in the diagram). For example, the output unit 104 displays a ranking so that each of the workers 20 can compete based on their scores.

[0053] Next, the operation of the information processing device 10 according to this embodiment will be described with reference to Figure 16. Figure 16 is a flowchart showing the flow of the score evaluation process for the worker 20. The CPU 11A of the information processing device 10 executes the score evaluation process shown in Figure 16. Each process in the information processing device 10 is executed by the CPU 11A functioning as a receiving unit 101, a generation unit 102, an evaluation unit 103, and an output unit 104. As mentioned above, the GPU 11B may also execute each process.

[0054] In step S100 of Figure 16, the CPU 11A receives the video captured by the camera 50.

[0055] In step S102, CPU 11A recognizes the joints of the human body in the image for each frame of the video.

[0056] In step S104, CPU 11A obtains the XY coordinates of each joint.

[0057] In step S106, CPU 11A converts each XY coordinate into time-series coordinate data.

[0058] In step S108, CPU 11A compares the time-series coordinate data of the expert and the time-series coordinate data of worker 20 using the DTW method. Specifically, it calculates the distance between each point in the two time-series coordinate data sets and calculates the path with the shortest distance.

[0059] In step S110, the CPU 11A quantifies the degree of difference in the time-series coordinate data of worker 20 using the calculated path distance.

[0060] In step S112, CPU 11A evaluates the difference using a score based on the aggregated difference values.

[0061] In step S114, the CPU 11A outputs a score for the evaluated worker 20's actions. Specifically, as shown in Figure 11, the CPU 11A displays the score on the score display section 71 of the video on the display device 40, which is a superimposed video of the current state and the example video. Note that the output of the score by the CPU 11A is not limited to screen display; it may also be output as audio.

[0062] In step S116, CPU 11A generates advice.

[0063] In step S118, the CPU 11A outputs the generated advice. Specifically, as shown in Figure 11, the CPU 11A displays the text related to the advice in the advice display section 72 of the display device 40. At the same time, it may also output a video that superimposes the actions of the expert and the worker 20.

[0064] In step S120, the CPU 11A determines whether the evaluation score is above or below the threshold value. If the CPU 11A determines that the evaluation score is above or below the threshold value (step S120: YES), the process terminates. On the other hand, if the CPU 11A determines that the evaluation score is not above or below the threshold value, i.e., below the threshold value (step S120: NO), the process returns to step S100. For example, if the threshold value is 90 points, if the evaluated score is 90 points or higher, the output unit 104 displays to the display device 40 that the stage of this task is cleared. If the evaluated score is below 90 points, the output unit 104 displays to the display device 40 that the worker should perform the task again, encouraging the worker to practice the task. In this way, by having the worker 20 repeat the task until their evaluation score exceeds the threshold value, the task is made easier to learn.

[0065] (remarks) In addition, the score evaluation process, which is executed by the CPU 11A or GPU 11B after reading the software (program) in the above embodiment, may also be executed by various processors other than the CPU and GPU. Examples of such processors include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits that are processors with circuit configurations specifically designed to execute specific processes, such as ASICs (Application Specific Integrated Circuits). Furthermore, the score evaluation process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0066] Furthermore, in the above embodiment, a display device 40 (specifically, a liquid crystal display, etc.) was assumed as one of the connection destinations for the input / output I / F of the information processing device 10, allowing the video to be viewed as an overlay, but the invention is not limited to this. For example, as an output device, the worker 20 may wear a VR headset and, in a state where they can visually look around 360 degrees in a virtual world, view a video that overlays the current video and the example video output by the output unit 104 after the work is completed. This allows the worker 20 to review the video after the work is completed, gaining an immersive experience as part of their learning training. Alternatively, as an output device, the worker 20 may wear an AR headset, which allows digital information to be overlaid on the real world's field of view, so the worker 20 can work in the real world while playing a video of a skilled worker's actions. In this case, since it may be dangerous to perform actual work while watching a video of a skilled worker, the AR headset as an output device of the information processing device 10 is intended to be used only for input, such as for initial practice videos and simulation of actions. In tasks that are subject to score evaluation by the information processing device 10, it is safer not to wear an AR headset that allows real-time viewing of the current situation and example videos. Therefore, it is preferable not to wear an AR headset when the evaluation unit 103 is evaluating the task. As described above, the output device related to the information processing device 10 is not limited to a liquid crystal display, and various outputs can be supported by using the information processing device 10 of this embodiment.

[0067] (summary) As described above, the information processing device 10 according to this embodiment includes a reception unit 101 that receives video footage of the worker 20's movements captured by the camera 50, a generation unit 102 that recognizes the joints of the worker 20's body in the video and generates time-series coordinate data, which is the time-series coordinate of each joint according to the movement, an evaluation unit 103 that evaluates the worker 20 with a score calculated based on the difference between the time-series coordinate data related to the worker 20's movements and the time-series coordinate data related to the movements of a skilled worker 20 who serves as a model, and an output unit 104 that outputs advice for the worker 20's movements corresponding to the score, along with the score. Therefore, according to the information processing device 10 according to this embodiment, the worker 20 can confidently work on learning the job.

[0068] To elaborate, according to the information processing device 10, workers 20 can objectively visualize and compare the correctness of their work procedures, the length of time required, etc., and since scores and advice are displayed, they can approach learning the work in a game-like manner. Furthermore, according to the information processing device 10, multiple workers 20 can compete for scores, which can stimulate their motivation to improve their learning. In this way, the information processing device 10 not only encourages workers 20 to actively improve their work, but also allows them to concentrate on their work because advice is not displayed during the work, enabling them to learn the work safely and with peace of mind. In addition to enabling safe and reproducible work learning, the gamification provided by the information processing device 10 reduces the burden on workers 20 in learning the work. [Explanation of Symbols]

[0069] 10 Information Processing Devices 11A CPU 11B GPU 12 ROM 13 RAM 14 Storage 15 Input / Output Interfaces 16 Communication I / F 17 Bus 20 workers 30 materials 33 Materials (work in progress) 35 Materials (Deliverables) 40 Display device 50 Cameras 71 Score display section 72. Advice display section 101 Reception Department 102 Generation part 103 Evaluation Department 104 Output section 105 Data Storage Unit

Claims

1. A reception unit that receives video footage of workers' movements captured by an imaging device, A generation unit that recognizes the joints of the worker's body in the aforementioned video and generates time-series coordinate data which is the time-series coordinate of each of the aforementioned joints corresponding to the aforementioned movements, An evaluation unit that evaluates based on a score compiled from the difference between the time-series coordinate data relating to the actions of the worker and the time-series coordinate data relating to the actions of a skilled worker who serves as a model, An output unit that outputs advice for the worker's actions corresponding to the score, along with the score, Information processing device including

2. The evaluation unit calculates the degree of difference between the time-series coordinate data of the skilled worker and the time-series coordinate data of the worker based on the distance between the time-series coordinate data of the two, and evaluates using the score, which is the sum of the degrees of difference. The information processing apparatus according to claim 1.

3. The output unit outputs a video in which the outlines of the skilled worker and each object, including the skilled worker and the objects involved in the skilled worker's actions, are represented in a single color over the video of the skilled worker's actions, and the current video in which the outlines of the worker and each object, including the worker and the objects involved in the worker's actions, are represented in a single color different from the single color mentioned above. The information processing apparatus according to claim 1.

4. When the worker's movement enters a predetermined position, the imaging device starts imaging. If the worker's movements intrude into another predetermined location, the output unit outputs at least one of the score, the advice, and the superimposed video. The information processing apparatus according to claim 3.

5. The reception unit receives video footage of the actions of multiple workers simultaneously captured by the imaging device. The evaluation unit evaluates each of the scores for each of the multiple individuals, The output unit outputs the results of a relative evaluation of the scores of the multiple people, reflecting the results in the advice. The information processing apparatus according to claim 1.

6. The system accepts video footage of the worker's movements captured by the imaging device. The system recognizes the joints of the worker's body in the aforementioned video and generates time-series coordinate data, which is the time-series coordinate of each joint corresponding to the aforementioned movement. The evaluation is performed using a score calculated based on the difference between the time-series coordinate data relating to the actions of the worker and the time-series coordinate data relating to the actions of a skilled worker who serves as a model. The system outputs advice regarding the worker's actions corresponding to the score, along with the score itself. An information processing method in which a computer performs the processing.

7. The system accepts video footage of the worker's movements captured by the imaging device. The system recognizes the joints of the worker's body in the aforementioned video and generates time-series coordinate data, which is the time-series coordinate of each joint corresponding to the aforementioned movement. The evaluation is performed using a score calculated based on the difference between the time-series coordinate data relating to the actions of the worker and the time-series coordinate data relating to the actions of a skilled worker who serves as a model. The system outputs advice regarding the worker's actions corresponding to the score, along with the score itself. An information processing program that instructs a computer to perform a task.

Images