Technology transfer tracing system and technology transfer tracing method

Abstract

To transfer a technology of an expert to posterity with higher accuracy.SOLUTION: A technological transfer trace system (100) for supporting transfer of technology of an expert to a worker includes: a state detection device (1) for detecting an attitude and a state of an object person; an imaging device (2) for imaging an ambient environment of the object person; and an information processing device (3) for generating operation model data about predetermined work of the expert on the basis of outputs preliminarily acquired from the state detection device and the imaging device in the predetermined work of the expert. An information processing device generates actual operation data corresponding to actual operation of the worker on the basis of the outputs acquired from the state detection device and the imaging device in the predetermined work of the worker, compares the operation model data with the actual operation data, and converts the operation model data into a format corresponding to the actual operation data to cause a display device (4) to display the actual operation data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a system and method for tracing technical transfer of a worker in a specific task such as a construction site. [Background technology]

[0002] For example, when an unskilled worker performs various tasks at a construction site, it is desirable for the worker to work under the direct guidance of an expert. However, an expert is not always present at the actual site, and there may be situations where the unskilled worker must work alone.

[0003] For example, Patent Document 1 discloses a motion teaching device using a head-mounted display device that allows the user to simultaneously view an artificial image superimposed on an external visual field image of the user. In Patent Document 1, a video of a specific motion of an expert is prepared in advance, and the video can be superimposed on the external visual field image of the user on the head-mounted display device.

[0004] Patent Document 2 discloses a work instruction system that instructs an unskilled worker on the actions of a skilled worker using a computer. In Patent Document 2, the actions of a skilled worker in a specified task using a tool are detected by a sensor and transferred to a processing device as action data. The processing device displays the action data three-dimensionally as a 3DCG virtual image on an output device. This allows an unskilled worker to perform the specified task while viewing the virtual image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 7-333550 [Patent Document 2] JP 2001-393188 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the images of the skilled worker prepared in advance did not necessarily match the field of view image of the actual worker, and from the perspective of passing on the skills of the skilled worker to future generations, there was room for improvement in accuracy.

[0007] The present invention has been made in consideration of the above points, and has an object to provide a technique transfer system that can transfer the techniques of experts to future generations with higher accuracy. [Means for solving the problem]

[0008] One embodiment of the technology transfer tracing system of the present invention is a technology transfer tracing system that supports the transfer of skills of an expert to a worker, and includes a state detection device that detects the posture and state of a subject, an imaging device that images the subject's surrounding environment, and an information processing device that generates action model data for a specified task of the expert based on output previously obtained from the state detection device and the imaging device, during the specified task of the expert, the information processing device generates actual action data corresponding to the actual action of the worker based on output previously obtained from the state detection device and the imaging device, compares the action model data with the actual action data, converts the action model data into a format corresponding to the actual action data, and displays it on a display device.

[0009] Moreover, a technology transfer tracing method according to one embodiment of the present invention is a technology transfer method for supporting the transfer of an expert's technology to a worker, and includes the steps of: generating motion model data for a specified task of the expert based on outputs obtained in advance from a state detection device which detects the posture and state of the subject and an imaging device which images the subject's surrounding environment; generating actual motion data corresponding to the actual motion of the worker based on outputs obtained in advance from the state detection device and the imaging device during the specified task of the worker; comparing the motion model data with the actual motion data; and converting the motion model data into a format corresponding to the actual motion data based on a result of the comparison between the motion model data and the actual motion data, and displaying the converted motion model data on a display device. Effect of the Invention

[0010] According to the present invention, the techniques of experts can be passed on to future generations with higher accuracy. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic configuration diagram showing an example of a technology transfer system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of the head mounted display device according to the embodiment. [Diagram 3] FIG. 13 is a conceptual diagram illustrating an example of a predetermined task according to the embodiment. [Figure 4] FIG. 11 is a diagram illustrating a flow of creating an expert motion model according to the embodiment; [Diagram 5] 1 is a diagram illustrating an image display flow of a head mounted display device according to an embodiment. FIG. [Figure 6] 1A to 1C are diagrams illustrating an example of an image display of a head mounted display device according to an embodiment. [Figure 7] 1A to 1C are diagrams illustrating an example of an image display of a head mounted display device according to an embodiment. [Figure 8] FIG. 2 is a diagram illustrating a hardware configuration of a computer for realizing an information processing device and a server device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the same reference numerals are used to designate corresponding elements in the various drawings.

[0013] FIG. 1 is a schematic diagram showing an example of a technology transfer tracing system according to an embodiment. A technology transfer tracing system 100 according to this embodiment (hereinafter simply referred to as the system 100) is for easily tracing the work of an expert when a worker at a specific work site such as a construction site performs a specific work. For example, the system 100 supports the transfer of a skill of an expert worker (which may be called an expert) to a relatively unskilled worker (hereinafter simply referred to as a worker). Note that, in this embodiment, a specific work at a construction site is described as an example, but the present invention is not limited to this and can be modified as appropriate. For example, the work site may be a construction site such as a road construction site or a care site at a care facility.

[0014] As shown in Fig. 1, the system 100 includes a head mounted display device 10 (hereinafter referred to as HMD device 10) that can be worn by a predetermined worker, and a server device 20 that systematically stores and manages motion model data related to a predetermined task of an expert. The HMD device 10 and the server device 20 can communicate with each other via a network. The HMD device 10 and the server device 20 may communicate with a network (such as the Internet) via wired and / or wireless (e.g., LTE (Long Term Evolution), NR (New Radio), Wi-Fi (registered trademark), etc.).

[0015] The HMD device 10 is configured with a display device of a type that is worn on a person's head. The HMD device 10 may be a non-transparent type that completely covers the eyes, or a transparent type display device that overlays an image (a virtual image such as a video) through a screen onto the real visual field. The HMD device 10 may also be called a wearable terminal. Details of the HMD device 10 will be described later.

[0016] Furthermore, subjects who wear the HMD device 10 include workers who work at construction sites, factories, and other such sites, as well as caregivers who care for people who require care (care recipients) living in care facilities. In this embodiment, the subjects may be called workers or skilled workers.

[0017] The server device 20 systematically stores motion model data related to predetermined tasks of skilled workers at a work site by task type and time series. The motion model data will be described later. The functions of the server device 20 may be built into the HMD device 10.

[0018] Fig. 2 is a block diagram showing an example of a functional configuration of a head mounted display device according to an embodiment. As shown in Fig. 2, the HMD device 10 includes a state detection device 1, an imaging device 2, an information processing device 3, a display device 4, and a helmet 5 (see Fig. 3). The various components of the state detection device 1, the imaging device 2, the information processing device 3, and the display device 4 are mounted on the helmet 5 worn by a worker.

[0019] The state detection device 1 detects the state and posture of a worker wearing an HMD device 10. The "state" of the worker represents the state of a given movement (e.g., walking), and the "posture" of the worker represents the current posture of the worker or a change thereto (e.g., standing posture, sitting posture, etc.).

[0020] Specifically, the condition detection device 1 includes a sensor 11, such as a gyro sensor or an acceleration sensor, that outputs detection values ​​corresponding to the subject's movements, and a memory 12. While the subject is working, the detection values ​​output from the sensor 11 are repeatedly stored in the memory 12.

[0021] Furthermore, the state detection device 1 outputs (transmits) the detection value of the sensor 11 to, for example, the imaging device 2 or the information processing device 3. The detection value may be output (transmitted) continuously or at predetermined time intervals. Although details will be described later, the detection value output from the sensor 11 is used when estimating the state and posture of the worker.

[0022] The imaging device 2 captures an image of the surrounding environment of the worker, and is attached to the body of the worker or any other location than the body of the worker. For example, when the imaging device 2 is attached to the body of the worker, the imaging device 2 is attached to the helmet 5 worn by the worker, as described above. When the imaging device 2 is attached to any other location than the body of the subject, the imaging device 2 may be installed at any location in a construction site, a factory, or a care facility, or may be mounted on an unmanned moving object such as a drone or a robot.

[0023] Moreover, the imaging device 2 includes a camera 21 and a memory 22. When the imaging device 2 is attached to the subject's body, the camera 21 is an action camera, a 360-degree camera, or the like, which captures images (still images or moving images) in the subject's field of view or around the subject, and stores the captured images in the memory 22. When the imaging device 2 is installed in an arbitrary location other than the subject's body, the camera 21 is a fixed surveillance camera installed in advance at a construction site, factory, or nursing facility, or a camera equipped on an unmanned mobile object, which captures images of the inside of a construction site, factory, or nursing facility including the subject, and stores the captured images in the memory 22.

[0024] The imaging device 2 reads out an image captured by the camera 21 from the memory 22, and outputs (transmits) the read image to the information processing device 3. Note that the output (transmission) of the image may be performed continuously or at predetermined time intervals.

[0025] The information processing device 3 is composed of a computer that generates predetermined data based on the output of the above-mentioned state detection device 1 and imaging device 2, and controls the display of the display device 3 described later. The information processing device 3 estimates the state, posture, etc. of the worker, and displays the estimated state and posture as an image superimposed on a screen. Specifically, the information processing device 3 includes, as functional blocks, an extraction unit 31, an estimation unit 32, an integration unit 33, and a comparison unit 34. The extraction unit 31, the estimation unit 32, the integration unit 33, and the comparison unit 34 may be collectively referred to as a control unit. Note that the information processing device 3 is not limited to these functional blocks, and may have other functional blocks.

[0026] The extraction unit 31 extracts and analyzes a predetermined object in an image transmitted from the imaging device 2 based on the image. Here, the object refers to a structure included in the image, such as a "floor", a "pillar", a "staircase", and a "handrail" as shown in FIG. 3. The extraction unit 31 may generate three-dimensional environmental data corresponding to the extracted object. The three-dimensional environmental data is three-dimensional data indicating the environment around the worker during a predetermined task, and includes data in which the above-mentioned objects are three-dimensionalized. The extraction unit 31 may extract objects corresponding to a "floor", a "pillar", a "staircase", a "handrail", and the like in an arbitrary three-dimensional space, and generate three-dimensional environmental data corresponding to each object for each time series at a predetermined time.

[0027] The estimation unit 32 generates three-dimensional skeleton data indicating the movement (posture) of the worker at a predetermined time based on the detection values ​​for a predetermined time transmitted from the state detection device 1 and the images for a predetermined time transmitted from the imaging device 2. That is, the estimation unit 32 estimates the state and posture of the worker at a predetermined time. Here, the three-dimensional skeleton data is three-dimensional data indicating the skeleton of the worker and the trajectory of the skeleton when the worker is performing a predetermined task in an arbitrary three-dimensional space. The three-dimensional skeleton data may be called three-dimensional movement data.

[0028] FIG. 3 is a conceptual diagram showing an example of the predetermined work according to the embodiment. As shown in FIG. 3, for example, a case is assumed in which a worker at a construction site is welding handrails of a staircase as the predetermined work. In this case, the estimation unit 32 generates 3D skeleton data showing the movement of the worker when he / she is crouching and welding. The estimation unit 32 may integrate data based on the state of the worker (state data) and data based on the posture of the worker (posture data) to generate 3D skeleton data for each time series at a predetermined time. Note that, in FIG. 3, welding of handrails of a staircase is described as an example of the predetermined work, but is not limited thereto and can be changed as appropriate. Other examples of the predetermined work include various tasks such as screwing and cutting sheet metal.

[0029] The integrating unit 33 integrates the above-mentioned object (or environmental three-dimensional data) and skeleton three-dimensional data to generate action model data. The action model data is generated based on the output of the state detection device 1 and the imaging device 2 by having a skilled person wearing the HMD device 10 perform a predetermined task in advance. The generated action model data may be stored in a storage area in the information processing device 3, or may be systematically stored (saved) for each task in a database of the server device 20 described later. The integrating unit 33 also generates actual action data based on the output acquired from the state detection device 1 and the imaging device 2 during the predetermined task of the worker.

[0030] The comparison unit 34 compares and collates the motion model data of the expert with the actual motion data of the worker, and converts the motion model data of the expert generated as described above into a format corresponding to the actual motion data. For example, the comparison unit 34 converts the motion model data into mixed reality data for display on the display device 4 using MR (Mixed Reality) technology.

[0031] Each functional configuration of the information processing device 3 may be realized by a server device 20, which will be described later.

[0032] The display device 4 is configured as a display provided at the lower front part of the helmet 5. The display device 4 is configured as, for example, a transmissive display, and is configured to superimpose the above-mentioned motion model data (mixed reality data) on the actual visual field image of the worker. The display device 4 may also display an image captured by the imaging device 2 as a video (video). Note that, in this embodiment, a case will be described in which the display device 4 is a part of a so-called head-mounted display device mounted on the helmet 5, but this is not limiting and appropriate modifications are possible. For example, the display device 4 may be configured as a mobile device such as a smartphone or a tablet terminal, or a personal computer.

[0033] As described above, the server device 20 has a database that systematically stores motion model data relating to predetermined tasks of skilled workers at a work site by task type and time series. For example, the motion model data for a specific task of an expert is (1) Name of work (e.g. welding, screwing, part positioning, etc.) (2) Work order (Step1, Step2, etc.) (3) Working time (hours, minutes, seconds, etc.) The items are set for each predetermined task and for each time series.

[0034] In this database, motion model data corresponding to each of a plurality of tasks is systematically stored. The motion model data is used as reference data (thresholds) to determine whether or not a specific task is being performed according to the correct procedure when an actual worker is performing the task. Note that the motion model data may include, in addition to the above (1)-(3), information on tips for the specific task and points requiring attention.

[0035] When unskilled workers perform various tasks at construction sites, it is desirable for them to work under the direct guidance of an expert. However, skilled workers are not always available at the actual site, and there may be situations where unskilled workers must work alone.

[0036] For example, until now, the typical method was for workers to learn how to do their jobs through trial and error, either by looking at a paper manual or by receiving verbal instructions from an expert during on-the-job training (OJT).

[0037] On the other hand, in recent years, the video that workers see on-site can be shared with a veteran (experienced) operator in a remote location, and the worker can follow instructions while working. There are also efforts to record the work process on video and understand the work content of the worker by watching the video.

[0038] However, with conventional technology, it takes time for a novice worker to reach the same level of work, speed, and quality as an expert. In addition, the occurrence of rework (redoing work) due to incorrect work execution is a factor that affects the subsequent process (schedule).

[0039] Therefore, the present inventors have conceived of the present invention with the aim of enabling even relatively unskilled workers to effectively learn a task without receiving direct instruction from an expert. For example, in this embodiment, a skilled worker is made to perform a predetermined task while wearing an HMD device 10, and motion model data of the skilled worker is generated and stored in a database. Then, the actual worker performs the task while wearing the HMD device 10. At this time, the HMD device 10 estimates the posture and motion (state) of the worker himself several seconds later from camera images and sensor data, and displays a correct motion image (virtual data) of the skilled worker on the display based on the motion model data of the skilled worker.

[0040] This allows an actual worker to see the movements (correct movements) of an expert in a specific task as an image superimposed on the real work space, and to work while checking the correct procedure of the expert. As a result, efficient and accurate work can be achieved. Furthermore, by having a worker actually trace the movements of an expert while working, the efficiency of learning the task and the speed of acquiring skills can be increased, and even a relatively unskilled worker can achieve work of the same quality as an expert. In other words, it is possible to pass on the skills of an expert to future generations with greater accuracy.

[0041] More specifically, the system 100 according to the present embodiment supports the transfer of the skilled worker's skills to the worker, and includes a state detection device 1 that detects the posture and state of the subject, an imaging device 2 that images the surrounding environment of the subject, and an information processing device 3 that generates motion model data related to the skilled worker's predetermined work based on the output previously acquired from the state detection device 1 and the imaging device 2 during the skilled worker's predetermined work. The information processing device 3 generates actual motion data corresponding to the worker's actual motion based on the output acquired from the state detection device 1 and the imaging device 2 during the worker's predetermined work, compares the motion model data with the actual motion data, converts the motion model data into a format corresponding to the actual motion data, and displays it on the display device 4. This allows the worker to work while checking the skilled worker's motion superimposed on the screen, thereby improving the worker's learning efficiency.

[0042] In this embodiment, the information processing device 3 compares the actual action data with the converted action model data to determine whether the predetermined task of the worker is correct. With this configuration, the difference between the actual action of the worker and the action of an expert can be clearly recognized, and the learning efficiency of the worker can be further improved.

[0043] In this embodiment, the information processing device 3 extracts objects related to the surrounding environment from the captured image of the surrounding environment of the subject, generates 3D skeleton data related to the skeleton of the subject based on the posture and state of the subject, and generates motion model data or actual motion data by integrating the objects and the 3D skeleton data. With this configuration, it is possible to generate data that estimates the actual motion of the worker from multiple data.

[0044] In this embodiment, an HMD device 10 (wearable terminal) that can be worn by a subject is configured by the state detection device 1, the imaging device 2, the information processing device 3, and the display device 4. By realizing these configurations in one device, it is possible to learn the movements of an expert with a simple configuration without interfering with the work of the worker.

[0045] In this embodiment, the system 100 further includes a server device 20 that systematically stores motion model data for each of a plurality of predetermined tasks. With this configuration, it is possible to systematically manage all tasks at a work site.

[0046] Next, a technology transfer tracing method according to the present embodiment will be described with reference to Figs. 4 to 7. Fig. 4 is a diagram illustrating an example of an expert's motion model creation flow according to the embodiment. Fig. 5 is a diagram illustrating an example of an image display flow of the HMD device according to the embodiment. Figs. 6 and 7 are diagrams illustrating examples of image display of the HMD device according to the embodiment. In the flow shown below, unless otherwise specified, the subject of the operation is the information processing device 3 (control unit). In addition, the flow shown below is merely an example, and other processes may be added as necessary. Also, the order of each process (step) can be changed as long as no contradiction occurs.

[0047] The technique transfer tracing method according to the present embodiment is a technique transfer method for supporting the transfer of techniques from an expert to a worker, and includes the following steps. (1) A step of generating motion model data for a specified task of an expert based on outputs acquired in advance from a state detection device 1 which detects the posture and state of a subject and an imaging device 2 which images the subject's surrounding environment (steps ST101-104 in FIG. 4). (2) A step of generating actual action data corresponding to an actual action of a worker based on outputs obtained from the state detection device 1 and the imaging device 2 during a predetermined task of the worker (steps ST201-204 in FIG. 5). (3) A step of comparing the motion model data with the actual motion data (step ST205 in FIG. 5). (4) A step of converting the behavior model data into a format corresponding to the actual behavior data based on the comparison result between the behavior model data and the actual behavior data, and displaying the converted behavior model data on a display device (step ST206 in FIG. 5). (5) A step of comparing the actual action data with the converted action model data to determine whether or not the predetermined work of the worker is correct (step ST207 in FIG. 5).

[0048] First, a flow for creating an expert motion model will be described. In this flow, expert motion model data corresponding to a predetermined task is generated and prepared in advance in the server device 20. In this case, it is assumed that the expert performs the predetermined task while wearing the HMD device 10.

[0049] 4, first, in step ST101, the information processing device 3 acquires various data. Specifically, the information processing device 3 acquires data on the posture and state of the skilled worker in a predetermined task from the state detection device 1, and acquires image data (captured image) corresponding to the surrounding environment of the skilled worker from the imaging device 2.

[0050] In the next step ST102, the information processing device 3 extracts a predetermined object (A) corresponding to the work site environment in time series from the captured images of the imaging device 2. For example, the information processing device 3 focuses on both the moving image of the skilled worker's body recorded in the imaging device 2 and the moving image showing the skilled worker's surroundings, and extracts what objects (e.g., pillars, beams, tools, etc.) are involved during the work and at what timing. Note that, in extracting the objects, it is possible to utilize an AI model that has previously learned each object of the work site.

[0051] In the next step ST103, the information processing device 3 generates three-dimensional skeleton data (B) estimating the state and posture of the skilled worker in the predetermined task. For example, the information processing device 3 focuses on both the moving image of the skilled worker's body recorded in the imaging device 2 and the moving image showing the skilled worker's surroundings, and estimates the skilled worker's three-dimensional posture (e.g. standing upright, squatting, lying down, etc.). The information processing device 3 also focuses on the previous moving image and data from the sensor 11, and estimates the skilled worker's three-dimensional state (e.g. walking, ascending, descending, falling, etc.).

[0052] In the next step ST104, the information processing device 3 generates motion model data (A+B) of an expert by integrating the object (A) and the three-dimensional skeleton data (B). The generated motion model data is stored in a predetermined storage area (the information processing device 3 or the server device 20).

[0053] Next, a screen display flow will be described. Steps ST201-204 in Fig. 5 are the same as steps ST101-104 in Fig. 4 except that "expert" is replaced with "operator" and "motion model data" is replaced with "actual motion data".

[0054] 5, first, in step ST201, the information processing device 3 acquires various data. Specifically, the information processing device 3 acquires data on the posture and state of the worker during a predetermined task from the state detection device 1, and acquires image data (captured image) corresponding to the surrounding environment of the worker from the imaging device 2.

[0055] In the next step ST202, the information processing device 3 extracts a predetermined object (C) corresponding to the work site environment in time series from the captured images of the imaging device 2. For example, the information processing device 3 focuses on both the moving image of the worker's body recorded in the imaging device 2 and the moving image showing the worker's surroundings, and extracts what object (e.g., pillar, beam, tool, etc.) is involved during the work and at what timing. Note that, in extracting the object, it is possible to utilize an AI model that has previously learned each object of the work site.

[0056] In the next step ST203, the information processing device 3 generates three-dimensional skeleton data (D) that estimates the state and posture of the worker during the specified task. For example, the information processing device 3 focuses on both the moving image of the worker's body recorded in the imaging device 2 and the moving image showing the worker's surroundings, and estimates the worker's three-dimensional posture (e.g., standing upright, crouching, lying down, etc.). The information processing device 3 also focuses on the previous moving image and data from the sensor 11, and estimates the worker's three-dimensional state (e.g., walking, ascending, falling, etc.).

[0057] In the next step ST204, the information processing device 3 integrates the object (C) and the three-dimensional skeleton data (D) to generate actual movement data (C+D) of the worker.

[0058] In the next step ST205, the information processing device 3 reads out the motion model data of the expert from the server device 20, and compares and collates it with the actual motion data of the worker.

[0059] In the next step ST206, the information processing device 3 converts the expert's motion model data according to the worker's state (actual motion data) and displays it superimposed on the screen. The work sites of the expert and the worker do not necessarily coincide, and by reflecting the actual worker's motion in the expert's motion model data, it is possible to improve the reproducibility of the teaching data (motion model data after conversion) in the real space.

[0060] In the next step ST207, the information processing device 3 displays the comparison result between the converted action model data and the actual action data in a superimposed manner on the screen. This provides feedback on whether the worker's action matches that of the expert, and enables the worker to visually recognize the accuracy of the worker's action. For example, the information processing device 3 may determine whether the work is based on a correct action based on whether the amount of deviation on the screen between the converted action model data and the actual action data is within a predetermined threshold.

[0061] According to such a screen display flow, for example, as shown in FIG. 6 and FIG. 7, an image based on the actual motion data of the worker is displayed on the screen with a solid line, and the converted motion model data of the expert is displayed with a two-dot chain line. The work sequence and the work time are displayed in the upper left of the screen. Also, as shown in FIG. 7, if the deviation amount on the screen between the converted motion model data and the actual motion data is within a predetermined threshold, the work is based on a correct motion and a message of "OK!" is displayed. For example, if the deviation amount on the screen between the converted motion model data and the actual motion data exceeds a predetermined threshold, the work is not based on a correct motion and a message of "NG!" may be displayed. Such a message (alert) may not only be displayed on the screen, but may also be notified by voice. Also, the information processing device 3 may control the image display so that, when the work in Step 1 in FIG. 6 is judged to be OK, the process proceeds to the next Step 2 shown in FIG. 7.

[0062] As described above, according to this embodiment, previously acquired motion model data for a specific task of an expert is converted to match the visual space of an actual worker and displayed on a screen, thereby making it possible to improve the learning efficiency of the actual worker.

[0063] 8 is a diagram showing an example of a computer for realizing the information processing device 3 or the server device 20. For convenience of explanation, the information processing device 3 and the server device 20 will be described using the same reference numerals.

[0064] 8 includes a processor 71, a memory 72, a storage device 73, a reader 74, a communication interface 75, and an input / output interface 76. The processor 71, the memory 72, the storage device 73, the reader 74, the communication interface 75, and the input / output interface 76 are connected to each other via a bus 77.

[0065] The processor 71 is configured with a single processor, a multiprocessor, a multicore, etc. The processor 71 also reads out a program stored in a storage device 73 and uses a memory 72 to execute a program describing the procedures of the flowcharts shown in Figures 4 and 5, thereby providing some or all of the functions of the control unit (e.g., the extraction unit 31, the estimation unit 32, the integration unit 33, and the control unit 34).

[0066] The memory 72 is a semiconductor memory or the like, and may include a RAM (Random Access Memory) area and a ROM (Read Only Memory) area.

[0067] The storage device 73 is a semiconductor memory such as a hard disk or a flash memory, or an external storage device.

[0068] The reader 74 accesses the removable storage medium 78 in accordance with instructions from the processor 71. The removable storage medium 78 is realized by a semiconductor device, a medium where information is input and output by magnetic action, a medium where information is input and output by optical action, or the like. The semiconductor device is a Universal Serial Bus (USB) memory or the like. The medium where information is input and output by magnetic action is a magnetic disk or the like. The medium where information is input and output by optical action is a Compact Disc (CD)-ROM, a Digital Versatile Disc (DVD), a Blu-ray Disc (Blu-ray is a registered trademark), or the like.

[0069] The communication interface 75 communicates with other devices in accordance with instructions from the processor 71. For example, the information processing device 3 collects information from the state detection device 1, the imaging device 2, and the server device 20 via the communication interface 75.

[0070] The input / output interface 76 is, for example, an interface between an input device and an output device. The input device is a device such as a keyboard, a mouse, or a touch panel that receives instructions from a subject or an administrator who checks the mixed reality data, the augmented reality data, or the virtual reality data. The output device is a display device such as a display, or an audio device such as a speaker.

[0071] Each program according to the embodiment is provided to the information processing device 3 in the following form, for example. (1) Pre-installed in storage device 73. (2) Provided by a removable storage medium 78. (3) Provided from a server such as a program server.

[0072] 8 is an example, and the embodiment is not limited thereto. For example, a part of the above-mentioned configuration may be deleted, or a new configuration may be added. In another embodiment, for example, a part or all of the functions of the above-mentioned information processing device 3 may be implemented as hardware such as an FPGA, a System-on-a-Chip (SoC), an Application Specific Integrated Circuit (ASIC), or a PLD.

[0073] The above describes the embodiments. However, the embodiments are not limited to the above embodiments, and should be understood to include various modified and alternative forms of the above embodiments. For example, it will be understood that the various embodiments can be embodied by modifying the components without departing from the spirit and scope of the embodiments. It will also be understood that various embodiments can be implemented by appropriately combining multiple components disclosed in the above-mentioned embodiments. Furthermore, it will be understood by those skilled in the art that various embodiments can be implemented by deleting some components from all the components shown in the embodiments, or by adding some components to the components shown in the embodiments. [Explanation of symbols]

[0074] 100 Technology Tracing System 1. Status detection device 2. Imaging device 3. Information processing equipment 4 Display device 5. Helmet 10 Head-mounted display device (wearable device) 11 Sensors 12. Memory 21 Camera 22 Memory 31 Extraction part 32 Estimation part 33 Integration Department 34 Comparison section

Claims

1. A technology transfer tracing system that supports the transfer of skills of an expert to a worker, A state detection device that detects the posture and state of a subject; an imaging device that captures an image of the subject's surroundings; an information processing device for extracting objects related to the expert's surrounding environment from an image of the subject's surrounding environment previously acquired from the imaging device during a predetermined task by the expert, generating three-dimensional skeleton data related to the skeleton of the expert based on the posture and state of the subject previously acquired from the state detection device, and integrating the objects related to the expert's surrounding environment and the three-dimensional skeleton data related to the skeleton of the expert to generate motion model data related to the predetermined task by the expert, the information processing device extracts objects related to the worker's surrounding environment from an image of the subject's surrounding environment acquired from the imaging device during the specified task of the worker, generates three-dimensional skeletal data related to the worker's skeleton based on the posture and state of the subject acquired from the state detection device, integrates the objects related to the worker's surrounding environment and the three-dimensional skeletal data related to the worker's skeleton to generate actual movement data corresponding to the actual movement of the worker, compares the movement model data with the actual movement data, converts the movement model data into a format corresponding to the actual movement data, and displays it on a display device.

2. 2. The technology transfer tracing system according to claim 1, wherein the information processing device compares the actual action data with the converted action model data to determine whether the predetermined work of the worker is correct.

3. The technology transfer tracing system according to claim 1 or 2, wherein the state detection device, the imaging device, the information processing device, and the display device constitute a wearable terminal that can be worn by a subject person.

4. 4. The technology transfer tracing system according to claim 1, further comprising a server device that systematically stores said behavior model data for each of said plurality of predetermined operations.

5. A technique transfer tracing method for supporting the transfer of a technique of an expert to a worker, comprising the steps of: extracting an object related to the surrounding environment of the expert from a captured image of the surrounding environment of the expert, the captured image having been acquired in advance from an imaging device that captures an image of the surrounding environment of the expert, during a predetermined task performed by the expert; generating three-dimensional skeleton data relating to the skeleton of the expert based on a posture and a state of the expert, the posture and the state of the expert being acquired in advance from a state detection device that detects a posture and a state of the expert during a predetermined task performed by the expert; A step of integrating objects related to the surrounding environment of the skilled person and three-dimensional skeleton data related to the skeleton of the skilled person to generate motion model data related to the predetermined task of the skilled person; extracting an object related to a surrounding environment of the worker from a captured image of the surrounding environment of the subject acquired from the imaging device during the predetermined task of the worker; generating three-dimensional skeleton data regarding a skeleton of the worker based on a posture and a state of the subject acquired from the state detection device during the predetermined task of the worker; generating actual movement data corresponding to an actual movement of the worker by integrating objects related to the surrounding environment of the worker and three-dimensional skeleton data related to the skeleton of the worker; comparing the motion model data with the actual motion data; a step of converting the behavior model data into a format corresponding to the actual behavior data based on a comparison result between the behavior model data and the actual behavior data, and displaying the converted behavior model data on a display device; The above-mentioned method for tracing technology transfer is executed by an information processing device.

6. 6. The technology transfer tracing method according to claim 5, further comprising the step of: comparing the actual action data with the converted action model data to determine whether the predetermined work performed by the worker is correct.

Images